# Learning to Prompt with Text Only Supervision for Vision-Language Models

**Authors:** Muhammad Uzair Khattak, Muhammad Ferjad Naeem, Muzammal Naseer, Luc Van Gool, Federico Tombari  
**arXiv:** [2401.02418](https://arxiv.org/abs/2401.02418)  
**Date:** 2024-01-04  
**Summary page:** https://www.summarizepaper.com/en/paper/2401.02418v1/

## Key points

- Challenges of adapting foundational vision-language models like CLIP for downstream tasks while maintaining generalization abilities
- Learning prompts using visual information requires labeled data and struggles to generalize to new datasets
- Generating class descriptions from large language models (LLMs) and performing prompt ensembling can result in class-specific prompts that cannot be transferred to other classes
- Proposed method combines the strengths of both approaches by learning prompts using only text data derived from LLMs
- Training approach allows prompts to extract rich contextual knowledge from LLM data, enabling zero-shot transfer of prompts to new classes and datasets
- Claimed as the first work to learn generalized prompts using text-only data
- Evaluation on four benchmarks demonstrates improvements over prior ensembling works and remains competitive with methods utilizing labeled images
- Ablative analysis on understanding ProText prompts and average confidence scores obtained from ProText logits trained on ImageNet-1k text data when applied to cross-datasets show increased performance compared to CLIP
- Introduces a promising approach for adapting vision-language models by learning prompts using only text data, potentially improving generalization capabilities while reducing reliance on labeled images.

## Summary

This paper discusses the challenges of adapting foundational vision-language models like CLIP for downstream tasks while maintaining their generalization abilities. One approach is to learn prompts using visual information; however, this often requires labeled data and struggles to generalize to new datasets. Another approach is to generate class descriptions from large language models (LLMs) and perform prompt ensembling; however, this can result in class-specific prompts that cannot be transferred to other classes. To address these limitations, the authors propose a novel method that combines the strengths of both approaches by learning prompts using only text data derived from LLMs. <br>
 Since supervised training of prompts is not possible without images, they develop a training approach that allows prompts to extract rich contextual knowledge from LLM data. This enables zero-shot transfer of prompts to new classes and datasets, potentially reducing the cost of LLM prompt engineering. The authors claim that their work is the first to learn generalized prompts using text-only data.<br>
They evaluate their method on four benchmarks and demonstrate improvements over prior ensembling works while remaining competitive with methods that utilize labeled images. In addition, the paper provides ablative analysis on understanding ProText prompts and presents average confidence scores obtained from ProText logits trained on ImageNet-1k text data when applied to cross-datasets. The results show increased performance compared to CLIP.<br>
Overall, this paper introduces a promising approach for adapting vision-language models by learning prompts using only text data. The proposed method has the potential to improve generalization capabilities while reducing reliance on labeled images.