# Detect Every Thing with Few Examples

**Authors:** Xinyu Zhang, Yuting Wang, Abdeslam Boularias  
**arXiv:** [2309.12969](https://arxiv.org/abs/2309.12969)  
**Date:** 2023-09-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.12969v1/

## Key points

- Open-set object detection goal: detect objects from unseen categories
- Recent advancements focus on open-vocabulary paradigm using vision-language backbones
- DE-ViT: new approach using vision-only DINOv2 backbones and example images for learning new categories
- DE-ViT transforms multi-classification tasks into binary classification tasks for more efficient detection
- Introduces novel region propagation technique for localization
- Outperforms state-of-the-art open-vocabulary method in COCO by 6.9 AP50, achieves 50 AP50 in novel classes
- Surpasses state-of-the-art few-shot method by 15 mAP on 10-shot and 7.2 mAP on 30-shot, and one-shot SoTA by 2.8 AP50 in COCO
- Outperforms state-of-the-art open-vocabulary method in LVIS by 2.2 mask AP, reaches a mask APr of 34.3
- Code available at https://github.com/mlzxy/devit
- Ablation studies conducted to analyze different aspects of DE-ViT's performance:
- Comparisons of different classification architectures and annotation types used to build prototypes
- Feature visualization comparing DINOv2 visual features with CLIP text features, suggesting images may be more promising than texts for representing classes.
- Overall, DE-ViT presents a novel approach achieving superior performance compared to existing methods

## Summary

In the field of open-set object detection, the goal is to detect objects that belong to categories not seen during training. Recent advancements in this area have focused on the open-vocabulary paradigm, which involves using vision-language backbones to represent categories with language. However, in this paper, the authors introduce a new approach called DE-ViT. <br>
DE-ViT is an open-set object detector that utilizes vision-only DINOv2 backbones and learns new categories through example images instead of language. To improve the general detection ability of DE-ViT, the authors propose transforming multi-classification tasks into binary classification tasks. This bypasses per-class inference and allows for more efficient detection. Additionally, they introduce a novel region propagation technique for localization.<br>
The performance of DE-ViT is evaluated on various benchmarks including COCO and LVIS datasets. In terms of COCO, DE-ViT outperforms the state-of-the-art (SoTA) open-vocabulary method by 6.9 AP50 and achieves 50 AP50 in novel classes. It also surpasses the SoTA few-shot method by 15 mAP on 10-shot and 7.2 mAP on 30-shot, as well as the one-shot SoTA by 2.8 AP50.<br>
For LVIS, DE-ViT outperforms the SoTA open-vocabulary method by 2.2 mask AP and reaches a mask APr of 34.3.<br>
The authors provide code for DE-ViT at https://github.com/mlzxy/devit.<br>
In addition to these results, ablation studies are conducted to analyze different aspects of DE-ViT's performance. These studies include comparisons of different classification architectures and annotation types used to build prototypes. Furthermore, feature visualization is performed to compare DINOv2 visual features with CLIP text features. The visualization shows that while CLIP visual features exhibit excellent intra-class compactness, there is a significant gap between CLIP text and visual features. This suggests that using images to represent classes may be more promising than relying solely on texts.<br>
Overall, DE-ViT presents a novel approach to open-set object detection that achieves superior performance compared to existing methods.