Semi-parametric Image Synthesis
We present a semi-parametric approach to photographic image synthesis from semantic layouts. The approach combines the complementary strengths of parametric and nonparametric techniques. The nonparametric component is a memory bank of image segments constructed from a training set of images. Given a novel semantic layout at test time, the memory bank is used to retrieve photographic references that are provided as source material to a deep network. The synthesis is performed by a deep network that draws on the provided photographic material. Experiments on multiple semantic segmentation datasets show that the presented approach yields considerably more realistic images than recent purely parametric techniques. The results are shown in the supplementary video at https://youtu.be/U4Q98lenGLQ
PDF Abstract CVPR 2018 PDF CVPR 2018 AbstractCode
Datasets
Cityscapes
ADE20K
GTA5
Results from the Paper
| Task | Dataset | Model | Metric Name | Metric Value | Global Rank | Benchmark |
|---|---|---|---|---|---|---|
| Image-to-Image Translation | ADE20K-Outdoor Labels-to-Photos | SIMS | mIoU | 13.1 | # 6 | |
| Accuracy | 74.7% | # 2 | ||||
| FID | 67.7 | # 5 | ||||
| Image-to-Image Translation | Cityscapes Labels-to-Photo | SIMS | Per-pixel Accuracy | 75.5% | # 7 | |
| mIoU | 47.2 | # 12 | ||||
| FID | 49.7 | # 4 |
