# PromptKD: Unsupervised Prompt Distillation for Vision-Language Models

**Authors:** Zheng Li, Xiang Li, Xinyi Fu, Xing Zhang, Weiqiang Wang, Jian Yang  
**arXiv:** [2403.02781](https://arxiv.org/abs/2403.02781)  
**Date:** 2024-03-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2403.02781v1/

## Key points

- Authors Zheng Li, Xiang Li, Xinyi Fu, Xing Zhang, Weiqiang Wang, and Jian Yang introduce PromptKD for enhancing vision-language models (VLMs)
- PromptKD uses prompts as distillers to transfer knowledge from large teacher models to smaller target models
- Framework consists of two key stages: pre-training a large CLIP teacher model with domain-specific labels and aligning logits via KL divergence and learnable prompts in the subsequent stage
- Eliminates need for labeled data by leveraging unlabeled images within the domain
- Ability to perform unsupervised domain-specific prompt-driven knowledge distillation for CLIP
- Practical mechanism established for pre-storing text features as shared class vectors between teacher and student models
- Demonstrated effectiveness through experiments on 11 datasets across various domains

## Summary

, , , , 

In their paper titled "PromptKD: Unsupervised Prompt Distillation for Vision-Language Models," authors Zheng Li, Xiang Li, Xinyi Fu, Xing Zhang, Weiqiang Wang, and Jian Yang introduce an innovative approach to enhancing vision-language models (VLMs) such as CLIP for specific domains. Their work focuses on utilizing prompts not only for learning but also as effective distillers for transferring knowledge from larger teacher models to smaller target models. The proposed framework consists of two key stages. In the initial stage, a large CLIP teacher model is pre-trained using domain-specific (few-shot) labels. Leveraging the unique decoupled-modality characteristics of CLIP, the text features are pre-computed and stored as class vectors through the teacher text encoder. In the subsequent stage, these stored class vectors are shared between the teacher and student image encoders to calculate predicted logits. By aligning the logits of both models via KL divergence and utilizing learnable prompts, the student image encoder is encouraged to generate probability distributions similar to those of the teacher model. This process eliminates the need for labeled data, allowing the algorithm to leverage a vast amount of unlabeled images within the domain. One significant aspect of this framework is its ability to perform unsupervised domain-specific prompt-driven knowledge distillation for CLIP. Additionally, a practical mechanism is established for pre-storing text features as shared class vectors between teacher and student models. The effectiveness of PromptKD is demonstrated through extensive experiments on 11 datasets across various domains. The well-trained student image encoders and pre-stored text features are then utilized for inference purposes. This research represents a novel contribution in leveraging prompts for knowledge distillation in VLMs and showcases promising results in enhancing vision-language models without relying on labeled data sources.