# Describe Anything: Detailed Localized Image and Video Captioning

**Authors:** Long Lian, Yifan Ding, Yunhao Ge, Sifei Liu, Hanzi Mao, Boyi Li, Marco Pavone, Ming-Yu Liu, Trevor Darrell, Adam Yala, Yin Cui  
**arXiv:** [2504.16072](https://arxiv.org/abs/2504.16072)  
**Date:** 2025-04-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2504.16072v1/

## Key points

- The Describe Anything Model (DAM) was introduced for Detailed Localized Captioning (DLC), balancing local details with global context in image and video captions.
- DLC-SDP, a Semi-supervised Learning-based Data Pipeline, improved DLC data quality by leveraging segmentation datasets and unlabeled web images.
- DLC-Bench was introduced as a benchmark for evaluating DLC without relying on reference captions, using an attribute-based evaluation approach.
- DAM achieved state-of-the-art performance across 7 benchmarks for regional captioning granularity.
- Off-the-shelf Vision-Language Models (VLMs) like GPT-4o and LLaVA excel at global-level descriptions but struggle with detailed localized captions due to limitations in specifying regions accurately.

## Summary

, , , , 

The realm of vision-language models has long struggled with generating accurate and detailed descriptions for specific regions in images and videos. To address this challenge, the Describe Anything Model (DAM) was introduced as a specialized model for Detailed Localized Captioning (DLC). By incorporating a focal prompt and a localized vision backbone, DAM effectively balances local details with global context in image and video captions. Additionally, the development of DLC-SDP, a Semi-supervised Learning-based Data Pipeline, has improved the quality of DLC data by leveraging segmentation datasets and unlabeled web images. The introduction of DLC-Bench as a benchmark for evaluating DLC without relying on reference captions marked a significant advancement in the field. The attribute-based evaluation approach used in DLC-Bench overcomes limitations associated with reference-based scoring methods. Notably, DAM has achieved state-of-the-art performance across 7 benchmarks that cover various levels of granularity in regional captioning. The challenges associated with generating detailed localized descriptions using off-the-shelf Vision-Language Models (VLMs) were also discussed. While VLMs like GPT-4o and LLaVA excel at producing global-level image descriptions, they struggle with providing detailed localized captions due to limitations in specifying regions of interest accurately. Various approaches were explored to address this issue, including presenting only the region to the VLM through masking or cropping and overlaying markings for localization cues. Overall, the advancements made through DAM, DLC-SDP, and DLC-Bench have significantly contributed to enhancing the capabilities of vision-language models in generating detailed localized image and video captions. The discussions presented shed light on the challenges faced in this domain and highlight potential avenues for further research and development.