Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRA
AI-generated Key Points
- Text-to-image generation is a growing research area that aims to synthesize high-quality images from textual descriptions
- Recent advances have shown progress in generating fine-grain details and customized concepts with few example images
- Continual customization of text-to-image models is a challenge when new concepts arrive sequentially, leading to catastrophic forgetting
- C-LoRA (Continually Self-Regularized Low-Rank Adaptation) efficiently adapts to new concepts while preserving knowledge of past concepts through continual self-regularized low-rank adaptation in cross attention layers of Stable Diffusion
- C-LoRA outperforms several baselines for Continual Diffusion and achieves state-of-the-art performance in rehearsal-free continual learning for image classification
- C-LoRA induces only marginal additional parameter costs and requires no storage of user data for replay
- Text-to-image generation has potential applications such as generating product images for ecommerce websites and creating personalized avatars for virtual reality environments
- C LoRA enables continuous customization without requiring retraining on past concepts after each new concept is added
Authors: James Seale Smith, Yen-Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, Hongxia Jin
Abstract: Recent works demonstrate a remarkable ability to customize text-to-image diffusion models while only providing a few example images. What happens if you try to customize such models using multiple, fine-grained concepts in a sequential (i.e., continual) manner? In our work, we show that recent state-of-the-art customization of text-to-image models suffer from catastrophic forgetting when new concepts arrive sequentially. Specifically, when adding a new concept, the ability to generate high quality images of past, similar concepts degrade. To circumvent this forgetting, we propose a new method, C-LoRA, composed of a continually self-regularized low-rank adaptation in cross attention layers of the popular Stable Diffusion model. Furthermore, we use customization prompts which do not include the word of the customized object (i.e., "person" for a human face dataset) and are initialized as completely random embeddings. Importantly, our method induces only marginal additional parameter costs and requires no storage of user data for replay. We show that C-LoRA not only outperforms several baselines for our proposed setting of text-to-image continual customization, which we refer to as Continual Diffusion, but that we achieve a new state-of-the-art in the well-established rehearsal-free continual learning setting for image classification. The high achieving performance of C-LoRA in two separate domains positions it as a compelling solution for a wide range of applications, and we believe it has significant potential for practical impact.
Ask questions about this paper to our AI assistant
You can also chat with multiple papers at once here.
Assess the quality of the AI-generated content by voting
Why do we need votes?
Votes are used to determine whether we need to re-run our summarizing tools. If the count reaches -10, our tools can be restarted.
Similar papers summarized with our AI tools
Navigate through even more similar papers through atree representation
Look for similar papers (in beta version)
By clicking on the button above, our algorithm will scan all papers in our database to find the closest based on the contents of the full papers and not just on metadata. Please note that it only works for papers that we have generated summaries for and you can rerun it from time to time to get a more accurate result while our database grows.
Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.