# HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention

**Authors:** Shijie Geng, Jianbo Yuan, Yu Tian, Yuxiao Chen, Yongfeng Zhang  
**arXiv:** [2303.02995](https://arxiv.org/abs/2303.02995)  
**Date:** 2023-03-06  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.02995v1/

## Key points

- Authors Shijie Geng, Jianbo Yuan, Yu Tian, Yuxiao Chen, and Yongfeng Zhang introduce HiCLIP for large-scale contrastive vision-language pretraining
- HiCLIP leverages hierarchical structures for enhanced multimodal content understanding
- HiCLIP improves cross-modal alignment by integrating hierarchy-aware attentions into both visual and language branches
- HiCLIP allows for unsupervised hierarchy induction from images and texts layer-by-layer
- Qualitative analysis demonstrates unsupervised hierarchy induction during inference with HiCLIP
- Extensive quantitative experiments across various visual recognition and vision-language tasks highlight the advantages of HiCLIP

## Summary

In their paper titled "HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention," authors Shijie Geng, Jianbo Yuan, Yu Tian, Yuxiao Chen, and Yongfeng Zhang delve into the realm of large-scale contrastive vision-language pretraining (CLIP) and its impact on visual recognition and multimodal content understanding. <br>
 represents a significant advancement in bridging the gap between visual and textual semantics by leveraging hierarchical structures for enhanced multimodal content understanding. While  has shown efficiency in inference compared to other models due to its concise design and lighter cross-attention fusion layers, it falls short in explicitly capturing the hierarchical nature of high-level and fine-grained semantics present in images and texts. To address this limitation, the authors introduce , a novel approach that integrates hierarchy-aware attentions into both the visual and language branches of . This enhancement allows  to progressively uncover semantic hierarchies layer-by-layer from images and texts in an unsupervised manner. By incorporating hierarchical aggregation,  significantly improves cross-modal alignment, thereby enhancing vision-language understanding and reasoning. The authors demonstrate the effectiveness of  through qualitative analysis showcasing its unsupervised hierarchy induction during inference. Additionally, they conduct extensive quantitative experiments across various visual recognition and vision-language downstream tasks to highlight the advantages of their proposed model. Ultimately,< kd > HiCLIP</ kd > represents a significant advancement in bridging the gap between visual and textual semantics by leveraging hierarchical structures for enhanced multimodal content understanding.