# Scalable Diffusion Models with Transformers

**Authors:** William Peebles, Saining Xie  
**arXiv:** [2212.09748](https://arxiv.org/abs/2212.09748)  
**Date:** 2022-12-19  
**Summary page:** https://www.summarizepaper.com/en/paper/2212.09748v2/

## Key points

- William Peebles and Saining Xie explore a new class of diffusion models based on the transformer architecture.
- These models replace the commonly-used U-Net backbone with a transformer that operates on latent patches.
- The scalability of these Diffusion Transformers (DiTs) is analyzed through forward pass complexity measured by Gflops.
- DiTs with higher Gflops consistently have lower Fréchet Inception Distance (FID), indicating better image quality.
- The largest DiT-XL/2 models outperform all prior diffusion models on class-conditional ImageNet benchmarks.
- They achieve a state-of-the-art FID of 2.27 on the 256x256 benchmark.
- Architectural choices such as adaptive normalization layers and channel counts for convolutional layers can impact model performance.
- DiTs demonstrate excellent scalability properties and achieve state-of-the-art performance on image generation tasks.

## Summary

In their paper, "Scalable Diffusion Models with Transformers," William Peebles and Saining Xie explore a new class of diffusion models based on the transformer architecture. These models are used for generating high-quality images by replacing the commonly-used U-Net backbone with a transformer that operates on latent patches. The scalability of these Diffusion Transformers (DiTs) is analyzed through the lens of forward pass complexity, measured by Gflops. The authors find that DiTs with higher Gflops, achieved through increased transformer depth/width or an increased number of input tokens, consistently have lower Fréchet Inception Distance (FID), a metric used to evaluate the quality of generated images. In addition to their good scalability properties, the largest DiT-XL/2 models outperform all prior diffusion models on the class-conditional ImageNet 512x512 and 256x256 benchmarks. They achieve a state-of-the-art FID of 2.27 on the latter benchmark. The authors discuss various architectural choices made in diffusion models such as adaptive normalization layers and channel counts for convolutional layers which can impact model performance. Overall, this paper presents a novel approach to diffusion modeling using transformers and provides insights into how different architectural choices can impact model performance. Their results demonstrate that DiTs have excellent scalability properties and achieve state-of-the-art performance on image generation tasks.