# MaPLe: Multi-modal Prompt Learning

**Authors:** Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, Fahad Shahbaz Khan  
**arXiv:** [2210.03117](https://arxiv.org/abs/2210.03117)  
**Date:** 2022-10-06  
**Summary page:** https://www.summarizepaper.com/en/paper/2210.03117v1/

## Key points

- CLIP has strong generalization capabilities in downstream tasks but is sensitive to input text prompts and templates
- Recent adaptations have focused on learning prompts for fine-tuning, but may not fully optimize both language and vision branches simultaneously
- MaPLe enhances alignment between vision and language representations by promoting strong coupling between their respective prompts
- MaPLe encourages synergy between modalities and discourages independent uni-modal solutions
- MaPLe incorporates separate prompts at different early stages to capture stage-wise feature relationships and facilitate rich context learning
- MaPLe outperforms Co-CoOp with a significant absolute gain of 3.45% on novel classes and 2.72% overall harmonic-mean across 11 diverse image recognition datasets

## Summary

The pre-trained vision-language (V-L) model CLIP has shown strong generalization capabilities in downstream tasks. However, it is sensitive to the selection of input text prompts and prompt templates. Recent adaptations have focused on learning prompts for fine-tuning the model, but this may not fully optimize both language and vision branches simultaneously. To address this limitation, we introduce Multi-modal Prompt Learning (MaPLe), which enhances alignment between vision and language representations by promoting strong coupling between their respective prompts. This design encourages synergy between the two modalities and discourages independent uni-modal solutions. Furthermore, MaPLe incorporates separate prompts at different early stages to capture stage-wise feature relationships and facilitate rich context learning. We evaluate our approach on various tasks including generalization to novel classes, new target datasets, and unseen domain shifts. In comparison to the state-of-the-art method Co-CoOp, MaPLe demonstrates superior performance with a significant absolute gain of 3.45% on novel classes and 2.72% on overall harmonic-mean across 11 diverse image recognition datasets. The detailed implementation details highlight how MaPLe improves alignment between vision and language representations while achieving impressive results in various challenging tasks. The full paper can be accessed at http://arxiv.org/pdf/2210.03117v1 for further insights into this innovative approach in multi-modal prompt learning for enhanced vision-language models like CLIP.