# PALP: Prompt Aligned Personalization of Text-to-Image Models

**Authors:** Moab Arar, Andrey Voynov, Amir Hertz, Omri Avrahami, Shlomi Fruchter, Yael Pritch, Daniel Cohen-Or, Ariel Shamir  
**arXiv:** [2401.06105](https://arxiv.org/abs/2401.06105)  
**Date:** 2024-01-11  
**Summary page:** https://www.summarizepaper.com/en/paper/2401.06105v1/

## Key points

- Content creators strive to create personalized images that encompass specific elements such as location, style, and ambiance
- Existing personalization methods often compromise either the ability to personalize or the alignment with complex textual prompts
- Proposed approach called prompt-aligned personalization improves text alignment and enables creation of images with complex prompts
- Method ensures prompt alignment using additional score distillation sampling term and can accommodate multiple subjects or draw inspiration from reference images
- Approach compared quantitatively and qualitatively with existing baselines and state-of-the-art techniques without relying on pre-training on large-scale data
- Superior results demonstrated compared to baselines in various settings
- Approach liberates content creators from constraints associated with specific prompts and allows them to fully unleash potential of text-to-image models
- Text-to-image synthesis has made significant progress due to large-scale training on datasets like LAION-400m
- Approach utilizes pre-trained diffusion models, primarily Stable-Diffusion (SD), for experiments but also verified on a larger latent diffusion model variant
- Other related methods include text-based editing approaches using multimodal models like CLIP for guidance, Prompt-to-Prompt (P2P) for editing generated images by manipulating attention maps, instruction-guided image-to-image translation methods preserving image structure using reference attention maps or features extracted through inversion, and early personalization methods like Textual Inversion and DreamBooth tuning pre-trained text-to-image models to represent new subjects.
- Evaluation conducted using StableDiffusion (SD) as a baseline and comparison with state-of-the-art techniques
- Alignment with target prompt measured using CLIP-score and subject preservation assessed through CLIP feature similarity between input and generated images.
- Overall, approach offers refined solution to personalized image generation by optimizing for both prompt alignment and subject fidelity.

## Summary

Content creators strive to create personalized images that go beyond conventional text-to-image models by encompassing specific elements such as location, style, and ambiance. However, existing personalization methods often compromise either the ability to personalize or the alignment with complex textual prompts. To address this issue, we propose a new approach called prompt-aligned personalization which excels in improving text alignment and enables the creation of images with complex prompts. Our method ensures prompt alignment using an additional score distillation sampling term and can accommodate multiple subjects or draw inspiration from reference images. We compare our approach quantitatively and qualitatively with existing baselines and state-of-the-art techniques without relying on pre-training on large-scale data. Through qualitative and quantitative analysis, we demonstrate superior results compared to baselines in various settings. Our approach liberates content creators from constraints associated with specific prompts and allows them to fully unleash the potential of text-to-image models. <br><br>
<br>
Text-to-image synthesis has made significant progress in recent years due to large-scale training on datasets like LAION-400m. Our approach utilizes pre-trained diffusion models to extend their understanding to new subjects. We primarily use Stable-Diffusion (SD) for our experiments but also verify our method on a larger latent diffusion model variant.<br>
Other related methods include text-based editing approaches that rely on contrastive multimodal models like CLIP for guidance. Prompt-to-Prompt (P2P) was proposed as a way to edit generated images by manipulating attention maps in cross-attention layers.<br>
Furthermore, there are instruction-guided image-to-image translation methods that preserve image structure using reference attention maps or features extracted through inversion.<br>
Early personalization methods like Textual Inversion and DreamBooth tune pre-trained text-to-image models to represent new subjects by finding new soft word embeddings or calibrating model weights with existing words.<br><br>
<br>
We evaluate our method using StableDiffusion (SD) as a baseline and compare it with state-of-the-art techniques. We also measure alignment with the target prompt using CLIP-score and assess subject preservation through CLIP feature similarity between input and generated images. Overall, our approach offers a refined solution to personalized image generation by optimizing for both prompt alignment and subject fidelity. It allows content creators to create images that accurately depict specific subjects while maintaining alignment with textual prompts.