# Object-Centric Open-Vocabulary Image-Retrieval with Aggregated Features

**Authors:** Hila Levi, Guy Heller, Dan Levi, Ethan Fetaya  
**arXiv:** [2309.14999](https://arxiv.org/abs/2309.14999)  
**Date:** 2023-09-26  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.14999v1/

## Key points

- Field of open-vocabulary object-centric image retrieval
- Task involves retrieving images with specific objects based on text queries
- Increasing importance due to use of large datasets for practical applications
- Challenges in current systems
- Limitations of single global embedding per image
- Scalability challenges with incorporating local embeddings from detection pipelines
- Proposed approach for object-centric image retrieval
- Aggregating dense embeddings from CLIP into compact representation
- Combining scalability and object identification capabilities
- Demonstrated effectiveness through experiments on three datasets, showing up to 15 mAP point increase in accuracy
- Integration into large-scale retrieval framework
- Advantages in scalability and interpretability demonstrated

## Summary

, , , , 

In the field of open-vocabulary object-centric image retrieval, the task involves retrieving images that contain a specific object of interest based on an open-set text query. As the use of large image datasets becomes more common, efficiently solving this task has become increasingly important for various practical applications. These applications include targeted performance analysis of retrieved images using customized queries and hard example mining during training processes. Recent advancements in contrastive-based open vocabulary systems have led to significant breakthroughs in facilitating large-scale open-vocabulary image retrieval. However, these approaches typically rely on a single global embedding per image, which limits the system's ability to accurately retrieve images containing relatively small object instances. On the other hand, incorporating local embeddings from detection pipelines presents scalability challenges, making it impractical for retrieval from extensive databases. In response to these challenges, a simple yet effective approach to object-centric open-vocabulary image retrieval has been developed. This approach involves aggregating dense embeddings extracted from CLIP into a compact representation, combining the scalability of image retrieval pipelines with the object identification capabilities of dense detection methods. The effectiveness of this scheme has been demonstrated through experiments on three datasets, showing significantly improved results compared to global feature approaches with an increase in accuracy by up to 15 mAP points. Furthermore, this method has been integrated into a large-scale retrieval framework, showcasing its advantages in terms of scalability and interpretability. The authors Hila Levi, Guy Heller, Dan Levi, and Ethan Fetaya have presented their findings in their paper titled "Object-Centric Open-Vocabulary Image-Retrieval with Aggregated Features," published at BMVC 2023. This work contributes valuable insights and practical solutions to enhance object-centric image retrieval tasks in complex and diverse datasets.