Key-Locked Rank One Editing for Text-to-Image Personalization
AI-generated Key Points
⚠The license of the paper does not allow us to build upon its content and the key points are generated using the paper metadata rather than the full article.
- The paper presents a novel approach to personalize text-to-image models (T2I) by aligning user-provided visual concepts with high visual fidelity and creative control.
- T2I models offer flexibility in guiding the creative process through natural language, but personalizing them remains difficult.
- The proposed method, Perfusion, uses dynamic rank-1 updates to the underlying T2I model to address these challenges.
- Perfusion introduces a new mechanism that "locks" new concepts' cross-attention Keys to their superordinate category, which prevents overfitting and enables combining multiple personalized concepts in a single image.
- The authors develop a gated rank-1 approach that allows controlling the influence of learned concepts during inference time and combining multiple concepts efficiently.
- This runtime-efficient balancing of visual fidelity and textual alignment is achieved using a single 100KB trained model, which is five orders of magnitude smaller than the current state-of-the-art.
- The proposed method outperforms strong baselines in both qualitative and quantitative terms.
- Key locking leads to novel results compared to traditional approaches, allowing portraying personalized object interactions in unprecedented ways even in one shot settings.
- Perfusion can span different operating points across the Pareto front without additional training.
- In conclusion, Perfusion offers an effective solution for T2I personalization by addressing multiple hard challenges such as maintaining high visual fidelity while allowing creative control and keeping small model size.
Authors: Yoad Tewel, Rinon Gal, Gal Chechik, Yuval Atzmon
Abstract: Text-to-image models (T2I) offer a new level of flexibility by allowing users to guide the creative process through natural language. However, personalizing these models to align with user-provided visual concepts remains a challenging problem. The task of T2I personalization poses multiple hard challenges, such as maintaining high visual fidelity while allowing creative control, combining multiple personalized concepts in a single image, and keeping a small model size. We present Perfusion, a T2I personalization method that addresses these challenges using dynamic rank-1 updates to the underlying T2I model. Perfusion avoids overfitting by introducing a new mechanism that "locks" new concepts' cross-attention Keys to their superordinate category. Additionally, we develop a gated rank-1 approach that enables us to control the influence of a learned concept during inference time and to combine multiple concepts. This allows runtime-efficient balancing of visual-fidelity and textual-alignment with a single 100KB trained model, which is five orders of magnitude smaller than the current state of the art. Moreover, it can span different operating points across the Pareto front without additional training. Finally, we show that Perfusion outperforms strong baselines in both qualitative and quantitative terms. Importantly, key-locking leads to novel results compared to traditional approaches, allowing to portray personalized object interactions in unprecedented ways, even in one-shot settings.
Ask questions about this paper to our AI assistant
You can also chat with multiple papers at once here.
⚠The license of the paper does not allow us to build upon its content and the AI assistant only knows about the paper metadata rather than the full article.
Welcome to our AI assistant! Here are some important things to keep in mind:
- The assistant will only answer questions related to this specific paper.
- Please note that this is not a bot for casual chatting.
- If you want the answer in a language other than the language you chose for navigating the website, simply add "TRANSLATE IN LANGUAGE L" at the end of your query (replace "LANGUAGE L" with the language of your choice).
- For example, you could ask "Can you extract the most important aspect of the paper? TRANSLATE IN SPANISH".
- If you want to keep the history of your questions/answers you should create an account.
Assess the quality of the AI-generated content by voting
Why do we need votes?
Votes are used to determine whether we need to re-run our summarizing tools. If the count reaches -10, our tools can be restarted.
Similar papers summarized with our AI tools
Navigate through even more similar papers through atree representation
Look for similar papers (in beta version)
By clicking on the button above, our algorithm will scan all papers in our database to find the closest based on the contents of the full papers and not just on metadata. Please note that it only works for papers that we have generated summaries for and you can rerun it from time to time to get a more accurate result while our database grows.
Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.