# FABRIC: Personalizing Diffusion Models with Iterative Feedback

**Authors:** Dimitri von Rütte, Elisabetta Fedele, Jonathan Thomm, Lukas Wolf  
**arXiv:** [2307.10159](https://arxiv.org/abs/2307.10159)  
**Date:** 2023-07-19  
**Summary page:** https://www.summarizepaper.com/en/paper/2307.10159v1/

## Key points

- Machine learning is driving visual content generation
- Human feedback can enhance user experience and output quality
- The study focuses on integrating human feedback into diffusion-based text-to-image models
- FABRIC is a training-free approach that uses the self-attention layer to condition the diffusion process on feedback images
- The proposed approach improves generation results through iterative feedback and optimization of user preferences
- Opportunities for personalized content creation and customization are significant
- Two experimental settings for automatic evaluation of generative visual models are proposed and used to evaluate FABRIC, showing its superiority over baseline methods
- Related work in textual inversion and style transfer techniques is discussed for personalizing text-to-image diffusion models
- The research contributes to advancing generative visual models by incorporating iterative human feedback and providing a robust evaluation methodology
- Implications for personalized content creation and customization exist.

## Summary

In an era where machine learning is increasingly driving visual content generation, incorporating human feedback into generative models has the potential to enhance user experience and output quality. This study focuses on diffusion-based text-to-image models and explores strategies for integrating iterative human feedback into the generative process. The authors propose FABRIC, a training-free approach that leverages the self-attention layer in popular diffusion models to condition the diffusion process on a set of feedback images. To rigorously evaluate their approach, they introduce a comprehensive evaluation methodology that quantifies the performance of generative visual models integrating human feedback. The study demonstrates that generation results improve over multiple rounds of iterative feedback through exhaustive analysis, implicitly optimizing arbitrary user preferences. The proposed approach offers significant opportunities for personalized content creation and customization. The authors also propose two experimental settings for automatic evaluation of generative visual models over multiple rounds, and using these settings, they evaluate FABRIC and show its superiority over baseline methods. The study also discusses related work in textual inversion and style transfer techniques for personalizing text-to-image diffusion models. Textual inversion allows learning semantic text embeddings from images depicting a common subject or style, enabling the synthesis of photorealistic images with desirable features. However, this technique requires multiple images incorporating those features and additional training to learn the semantic embedding. Overall, this research contributes to advancing generative visual models by incorporating iterative human feedback and providing a robust evaluation methodology. The findings have implications for fields such as personalized content creation and customization.