# In-Context Learning Unlocked for Diffusion Models

**Authors:** Zhendong Wang, Yifan Jiang, Yadong Lu, Yelong Shen, Pengcheng He, Weizhu Chen, Zhangyang Wang, Mingyuan Zhou  
**arXiv:** [2305.01115](https://arxiv.org/abs/2305.01115)  
**Date:** 2023-05-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.01115v1/

## Key points

- The Prompt Diffusion framework enables in-context learning in diffusion-based generative models.
- It takes a pair of task-specific example images and text guidance to understand the underlying task and perform the same task on a new query image.
- The key innovation is the vision-language prompt that can model a wide range of vision-language tasks and a diffusion model that takes it as input.
- The diffusion model is trained jointly over six different tasks using these prompts, resulting in the first diffusion-based vision-language foundation model capable of in-context learning.
- The Prompt Diffusion model demonstrates high quality in-context generation on the trained tasks and generalizes effectively to new, unseen vision tasks with their respective prompts.
- Additionally, it shows compelling text guided image editing results.
- This framework aims to facilitate research into in-context learning for computer vision and has its code publicly available at https://github.com/Zhendong-Wang/Prompt-Diffusion.
- The authors are Zhendong Wang, Yifan Jiang, Yadong Lu, Yelong Shen, Pengcheng He, Weizhu Chen, Zhangyang Wang and Mingyuan Zhou.
- Their work titled "In Context Learning Unlocked for Diffusion Models" presents an innovative solution to enable in context learning for diffusion based generative models by leveraging vision language prompts.
- Overall this work represents significant progress towards developing more effective computer vision systems capable of performing complex tasks while also being able to learn from context.

## Summary

The Prompt Diffusion framework is a novel approach for enabling in-context learning in diffusion-based generative models. The model takes a pair of task-specific example images, such as depth from/to image and scribble from/to image, along with text guidance to understand the underlying task and perform the same task on a new query image following the text guidance. The key innovation of this framework is the vision-language prompt that can model a wide range of vision-language tasks and a diffusion model that takes it as input. The diffusion model is trained jointly over six different tasks using these prompts, resulting in the first diffusion-based vision-language foundation model capable of in-context learning. The Prompt Diffusion model demonstrates high quality in-context generation on the trained tasks and generalizes effectively to new, unseen vision tasks with their respective prompts. Additionally, it shows compelling text guided image editing results. This framework aims to facilitate research into in-context learning for computer vision and has its code publicly available at https://github.com/Zhendong-Wang/Prompt-Diffusion. The authors of this paper are Zhendong Wang, Yifan Jiang, Yadong Lu, Yelong Shen, Pengcheng He, Weizhu Chen, Zhangyang Wang and Mingyuan Zhou. Their work titled "In Context Learning Unlocked for Diffusion Models" presents an innovative solution to enable in context learning for diffusion based generative models by leveraging vision language prompts. Overall this work represents significant progress towards developing more effective computer vision systems capable of performing complex tasks while also being able to learn from context.