# eDiffi: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers

**Authors:** Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, Ming-Yu Liu  
**arXiv:** [2211.01324](https://arxiv.org/abs/2211.01324)  
**Date:** 2022-11-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2211.01324v1/

## Key points

- eDiffi is a text-to-image diffusion model that improves the synthesis process.
- It uses an ensemble of expert denoisers to enhance text alignment without compromising visual quality or computational cost.
- Specialized models are trained for different synthesis stages to address the issue of changing generation behavior throughout the iterative process.
- Various embeddings, including T5 text, CLIP text, and CLIP image embeddings, are used for conditioning.
- eDiffi offers a "paint-with-words" capability that allows users to select words in the input text and control the output image.
- Experimental results show that eDiffi outperforms previous large-scale text-to-image diffusion models on benchmark datasets.
- It handles long detailed descriptions better than methods like DALL·E 2 and Stable Diffusion.
- Increasing network depth or adding more experts does not significantly impact feed-forward evaluation time.
- eDiffi provides state-of-the-art performance in high-resolution image synthesis from text prompts while offering controllability and efficiency enhancements.

## Summary

, , , , 

The paper presents eDiffi, a text-to-image diffusion model that utilizes an ensemble of expert denoisers to improve the synthesis process. The authors observe that the generation behavior of text-to-image diffusion models changes throughout the iterative process, with early stages relying heavily on text prompts and later stages ignoring text conditioning. To address this issue, eDiffi trains specialized models for different synthesis stages, resulting in improved text alignment without compromising visual quality or computational cost. The model is trained using various embeddings for conditioning, including T5 text, CLIP text, and CLIP image embeddings. Additionally, eDiffi offers a unique "paint-with-words" capability that allows users to select words in the input text and paint them on a canvas to control the output image. Experimental results demonstrate that eDiffi outperforms previous large-scale text-to-image diffusion models on benchmark datasets. Furthermore, comparisons with other methods such as DALL·E 2 and Stable Diffusion highlight eDiffi's ability to handle long detailed descriptions better than these methods. The inference time of eDiffi is also evaluated based on model capacity, showing that increasing network depth or adding more experts does not significantly impact feed-forward evaluation time. In summary, offers state-of-the-art performance in high-resolution image synthesis from text prompts while providing controllability and efficiency enhancements.