# Zero-shot Referring Expression Comprehension via Structural Similarity Between Images and Captions

**Authors:** Zeyu Han, Fangrui Zhu, Qianru Lao, Huaizu Jiang  
**arXiv:** [2311.17048](https://arxiv.org/abs/2311.17048)  
**Date:** 2023-11-28  
**Summary page:** https://www.summarizepaper.com/en/paper/2311.17048v1/

## Key points

- Authors propose a novel zero-shot referring expression comprehension model
- Model leverages foundation models like ChatGPT and CLIP
- Goal is to model structural similarity between entities in images and texts
- Triplet-matching objective introduced to improve visual relationship understanding of VLA models
- Experimental results validate effectiveness on RefCOCO/+/g and Who's Waldo datasets
- Visual grounding performance increases by up to 19.5% over state-of-the-art zero-shot model on RefCOCO/+/g
- Zero-shot approach achieves comparable accuracy to fully supervised model on Who's Waldo dataset
- Implementation details provided for special cases of textual triplets without explicit subjects or single nouns
- Prompts can be divided into four parts: general instruction, supporting details, in-context learning examples, and task instruction
- Captions parsed into triplets using ChatGPT's in-context learning capability
- Bounding boxes used to represent entities in image for visual triplets construction
- Visual pairs generated using Cartesian product approach for all possible combinations of entities

## Summary

In this paper, the authors propose a novel zero-shot referring expression comprehension model that leverages powerful foundation models like ChatGPT and CLIP. The goal is to explicitly model the structural similarity between entities in both images and texts, and then find their correspondences by propagating the similarity from triplets to instances. To improve the visual relationship understanding of vision-language alignment (VLA) models, a novel triplet-matching objective is introduced to fine-tune them on a curated dataset containing abundant entity relationships. The experimental results on RefCOCO/+/g and Who's Waldo datasets validate the effectiveness of the proposed approach. The visual grounding performance increases by up to 19.5% over the state-of-the-art zero-shot model on RefCOCO/+/g. Additionally, on the more challenging Who's Waldo dataset, the zero-shot approach achieves comparable accuracy to the fully supervised model. The authors also provide implementation details for special cases of textual triplets where captions may lack explicit subjects or consist of single nouns. In such cases, blank spaces are filled with the subject string to ensure proper management by the grounding pipeline. Furthermore, they illustrate how prompts can be divided into four parts: general instruction, supporting details, in-context learning examples, and task instruction. Leveraging ChatGPT's in-context learning capability, captions can be parsed into triplets for further processing. For visual triplets construction, bounding boxes are used to represent entities in an image. Visual pairs are generated using a Cartesian product approach to consider all possible combinations of entities. Overall, this paper presents a comprehensive approach that addresses the challenges of zero-shot referring expression comprehension by utilizing foundation models' capabilities and incorporating fine-grained disentanglement and relationship understanding techniques. The experimental results demonstrate significant improvements in visual grounding performance and comparable accuracy to fully supervised models on challenging datasets.