Diffusion Guided Domain Adaptation of Image Generators
AI-generated Key Points
- The paper proposes a method for adapting a GAN generator to a new domain using text-to-image diffusion models as training objectives.
- Classifier-free guidance is used as a critic to enable generators to distill knowledge from large-scale text-to-image diffusion models, allowing them to efficiently shift into new domains indicated by text prompts without access to ground truth samples.
- The authors demonstrate the effectiveness and controllability of their method through extensive experiments, achieving high CLIP scores and significantly lower FID than prior work on short prompts, and outperforming the baseline qualitatively and quantitatively on long and complicated prompts.
- The proposed method incorporates large-scale pre-trained diffusion models and distillation sampling for text-driven image generator domain adaptation, giving quality previously beyond possible.
- The authors extend their work to 3D-aware style-based generators and DreamBooth guidance.
- Performance gains increase quickly as the text prompts grow longer, with the method generating images with much higher visual quality and fidelity in these experiments.
- Quantitative comparisons show that the models achieve significantly better FIDs than the baseline, competitive CLIP scores with better LPIPS scores, and capture all key constraints mentioned in long text prompts more effectively than the baseline.
- Overall, this work presents an innovative approach for adapting image generators to new domains using large scale pre trained diffusion models and distillation sampling guided by textual input.
Authors: Kunpeng Song, Ligong Han, Bingchen Liu, Dimitris Metaxas, Ahmed Elgammal
Abstract: Can a text-to-image diffusion model be used as a training objective for adapting a GAN generator to another domain? In this paper, we show that the classifier-free guidance can be leveraged as a critic and enable generators to distill knowledge from large-scale text-to-image diffusion models. Generators can be efficiently shifted into new domains indicated by text prompts without access to groundtruth samples from target domains. We demonstrate the effectiveness and controllability of our method through extensive experiments. Although not trained to minimize CLIP loss, our model achieves equally high CLIP scores and significantly lower FID than prior work on short prompts, and outperforms the baseline qualitatively and quantitatively on long and complicated prompts. To our best knowledge, the proposed method is the first attempt at incorporating large-scale pre-trained diffusion models and distillation sampling for text-driven image generator domain adaptation and gives a quality previously beyond possible. Moreover, we extend our work to 3D-aware style-based generators and DreamBooth guidance.
Ask questions about this paper to our AI assistant
You can also chat with multiple papers at once here.
Welcome to our AI assistant! Here are some important things to keep in mind:
- The assistant will only answer questions related to this specific paper.
- Please note that this is not a bot for casual chatting.
- If you want the answer in a language other than the language you chose for navigating the website, simply add "TRANSLATE IN LANGUAGE L" at the end of your query (replace "LANGUAGE L" with the language of your choice).
- For example, you could ask "Can you extract the most important aspect of the paper? TRANSLATE IN SPANISH".
- If you want to keep the history of your questions/answers you should create an account.
Assess the quality of the AI-generated content by voting
Why do we need votes?
Votes are used to determine whether we need to re-run our summarizing tools. If the count reaches -10, our tools can be restarted.
Similar papers summarized with our AI tools
Navigate through even more similar papers through atree representation
Look for similar papers (in beta version)
By clicking on the button above, our algorithm will scan all papers in our database to find the closest based on the contents of the full papers and not just on metadata. Please note that it only works for papers that we have generated summaries for and you can rerun it from time to time to get a more accurate result while our database grows.
Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.