# Betrayed by Captions: Joint Caption Grounding and Generation for Open Vocabulary Instance Segmentation

**Authors:** Jianzong Wu, Xiangtai Li, Henghui Ding, Xia Li, Guangliang Cheng, Yunhai Tong, Chen Change Loy  
**arXiv:** [2301.00805](https://arxiv.org/abs/2301.00805)  
**Date:** 2023-01-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2301.00805v1/

## Key points

- Authors: Jianzong Wu, Xiangtai Li, Henghui Ding, Xia Li, Guangliang Cheng, Yunhai Tong, Chen Change Loy
- Goal: Improve instance-level open vocabulary segmentation without mask annotations
- Utilize image captions to identify instances of novel categories
- End-to-end framework centered around caption grounding and generation
- Joint Caption Grounding and Generation (CGG) framework built upon Mask Transformer baseline
- Unique grounding loss mechanism for multi-modal feature alignments
- Lightweight caption generation head for supplementary supervision
- Significant improvements in segmentation performance for novel classes
- 6.8% increase in mean Average Precision (mAP) on novel classes without additional caption data
- PQ improvements exceeding 15% for novel classes on OSPS benchmark

## Summary

In their research paper titled "Betrayed by Captions: Joint Caption Grounding and Generation for Open Vocabulary Instance Segmentation," authors Jianzong Wu, Xiangtai Li, Henghui Ding, Xia Li, Guangliang Cheng, Yunhai Tong, and Chen Change Loy explore the realm of instance-level open vocabulary segmentation. Their main goal is to improve a segmenter's ability to identify novel categories at the instance level without relying on mask annotations. The study introduces a novel approach that utilizes image captions to aid in this task. By utilizing the vast number of object nouns present in captions, the researchers aim to uncover instances of previously unseen classes. Unlike existing methods that use pretrained caption models or complex pipelines with large caption datasets, their proposed solution takes an alternative route by offering an end-to-end framework centered around caption grounding and generation. At the core of their methodology lies the development of a joint Caption Grounding and Generation (CGG) framework built upon a Mask Transformer baseline. This framework incorporates a unique grounding loss mechanism that facilitates explicit and implicit multi-modal feature alignments. Additionally, a lightweight caption generation head is designed to provide supplementary caption supervision. Through extensive experimentation on the COCO dataset under two distinct settings - Open Vocabulary Instance Segmentation (OVIS) and Open Set Panoptic Segmentation (OSPS) - the authors showcase the effectiveness of their CGG framework. Results demonstrate significant improvements in segmentation performance for novel categories, with a notable 6.8% increase in mean Average Precision (mAP) on novel classes without additional caption data. Moreover, their method achieves PQ improvements exceeding 15% for novel classes on the OSPS benchmark across various configurations. In summary,"Betrayed by Captions" presents an innovative approach to open vocabulary instance segmentation that leverages image captions to enhance model performance in identifying novel categories without relying on mask annotations. The CGG framework developed by Wu et al. showcases promising results in significantly improving segmentation accuracy for previously unseen classes within diverse visual datasets like COCO.