# High-Resolution Image Synthesis with Latent Diffusion Models

**Authors:** Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Björn Ommer  
**arXiv:** [2112.10752](https://arxiv.org/abs/2112.10752)  
**Date:** 2021-12-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2112.10752v1/

## Key points

- The paper presents a novel approach to image synthesis using diffusion models (DMs)
- DMs decompose the image formation process into a sequential application of denoising autoencoders
- DMs achieve state-of-the-art results on various tasks, including semantic scene synthesis, super-resolution, and image inpainting
- To address limitations such as optimization requiring hundreds of GPU days and expensive inference due to sequential evaluations, the authors propose applying DMs in the latent space of powerful pretrained autoencoders
- This approach allows for training on limited computational resources while retaining quality and flexibility
- The authors introduce cross-attention layers into the model architecture to turn diffusion models into powerful and flexible generators for general conditioning inputs such as text or bounding boxes
- The proposed latent diffusion models (LDMs) achieve a new state-of-the-art performance for image inpainting and highly competitive results on various tasks while significantly reducing computational requirements compared to pixel-based DMs.
- The code is available at https://github.com/CompVis/latent-diffusion.

## Summary

The paper titled "High-Resolution Image Synthesis with Latent Diffusion Models" presents a novel approach to image synthesis using diffusion models (DMs) that achieve state-of-the-art results on various tasks, including semantic scene synthesis, super-resolution, and image inpainting. DMs operate by decomposing the image formation process into a sequential application of denoising autoencoders, allowing for a guiding mechanism to control the image generation process without retraining. To address limitations such as optimization requiring hundreds of GPU days and expensive inference due to sequential evaluations, the authors propose applying DMs in the latent space of powerful pretrained autoencoders. This approach allows for training on limited computational resources while retaining quality and flexibility. The authors demonstrate that training diffusion models on such a representation enables reaching a near-optimal point between complexity reduction and detail preservation, greatly boosting visual fidelity. Moreover, the authors introduce cross-attention layers into the model architecture to turn diffusion models into powerful and flexible generators for general conditioning inputs such as text or bounding boxes. This innovation makes high-resolution synthesis possible in a convolutional manner. The proposed latent diffusion models (LDMs) achieve a new state-of-the-art performance for image inpainting and highly competitive results on various tasks while significantly reducing computational requirements compared to pixel-based DMs. The code is available at https://github.com/CompVis/latent-diffusion. Overall, this paper presents an innovative approach to image synthesis that addresses some of the limitations of existing methods by leveraging pretrained autoencoders and introducing cross-attention layers. The proposed LDMs achieve impressive results on various tasks while reducing computational requirements compared to previous methods.