# Picture that Sketch: Photorealistic Image Generation from Abstract Sketches

**Authors:** Subhadeep Koley, Ayan Kumar Bhunia, Aneeshan Sain, Pinaki Nath Chowdhury, Tao Xiang, Yi-Zhe Song  
**arXiv:** [2303.11162](https://arxiv.org/abs/2303.11162)  
**Date:** 2023-03-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.11162v2/

## Key points

- Novel approach for generating photorealistic images from abstract sketches
- Works with free-hand human sketches, accessible to amateurs
- Decoupled encoder-decoder training paradigm using StyleGAN
- Autoregressive sketch mapper bridges abstraction gap between sketch and photo
- Specific designs including fine-grained discriminative loss and partial-aware sketch augmentation strategy
- Fine-grained sketch-based image retrieval with superior performance compared to state-of-the-art methods
- Precise semantic editing with consistent local changes in generated images
- Fine-grained control over appearance features through multi-modal generation
- Outperforms existing state-of-the-art approaches in terms of generating photorealistic images from abstract sketches
- Detailed results and comparisons provided with metrics such as LPIPS and FID scores

## Summary

, , , , 

This paper presents a novel approach for generating photorealistic images from abstract sketches. Unlike previous methods that require specific edgemap-like sketches, this approach works with free-hand human sketches, making it accessible to amateurs. The key contribution of this work is a decoupled encoder-decoder training paradigm, where the decoder is a StyleGAN trained on photos only, ensuring that the generated results are always photorealistic. To bridge the abstraction gap between sketch and photo, the authors propose an autoregressive sketch mapper trained on sketch-photo pairs. This mapper maps a sketch to the latent space of the pre-trained StyleGAN to generate realistic images. To address the abstract nature of human sketches, the authors introduce specific designs including a fine-grained discriminative loss and a partial-aware sketch augmentation strategy. The paper also demonstrates several downstream applications enabled by their generation model. One application is fine-grained sketch-based image retrieval, where they convert the task into an image-based retrieval task and achieve superior performance compared to state-of-the-art methods. Another application is precise semantic editing, where users can modify specific regions of an input sketch and observe consistent local changes in the generated images. Additionally, the proposed method allows for fine-grained control over appearance features through multi-modal generation. By replacing medium or fine-level latent codes with random vectors, users can explore different color variations and appearance features in the generated images. Overall, extensive experiments show that this method outperforms existing state-of-the-art approaches in terms of generating photorealistic images from abstract sketches. The authors provide detailed results and comparisons with other methods in terms of metrics such as LPIPS and FID scores. In conclusion, this paper presents a novel approach for generating photorealistic images from abstract sketches without requiring specific edgemap-like inputs. The proposed method achieves superior performance compared to existing state-of-the-art approaches and enables various downstream applications such as fine-grained image retrieval and precise semantic editing.