# DETRs with Collaborative Hybrid Assignments Training

**Authors:** Zhuofan Zong, Guanglu Song, Yu Liu  
**arXiv:** [2211.12860](https://arxiv.org/abs/2211.12860)  
**Date:** 2022-11-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2211.12860v5/

## Key points

- Authors address the issue of sparse supervision in DETR models caused by too few positive samples assigned during training
- Proposed training scheme called Co-DETR (Collaborative Hybrid Assignments Training) to enhance learning ability of DETR-based detectors
- Co-DETR improves feature learning in encoder and attention learning in decoder through two main components: collaborative hybrid assignments training and customized positive queries generation
- Collaborative hybrid assignments training involves training multiple parallel auxiliary heads supervised by one-to-many label assignments such as ATSS and Faster RCNN
- Customized positive queries are generated by extracting positive coordinates from these auxiliary heads, improving efficiency of training positive samples in the decoder
- During inference, auxiliary heads are discarded with no additional parameters or computational cost to original detector
- Co-DETR eliminates need for handcrafted non-maximum suppression (NMS)
- Evaluated on various DETR variants achieving state-of-the-art results on COCO val dataset with improvement from 58.5% to 59.5%
- When incorporated with ViT backbone, achieves impressive results of 66.0% AP on COCO test dev dataset and 67.9% AP on LVIS val dataset with significantly fewer model sizes
- Co-DETR presents an effective solution for improving feature learning and attention learning in DETR-based detectors while achieving state-of-the-art performance on various benchmark datasets

## Summary

In this paper, the authors address the issue of sparse supervision in DETR (Detection Transformer) models caused by too few positive samples assigned during training. They propose a novel training scheme called Co-DETR (Collaborative Hybrid Assignments Training) to enhance the learning ability of DETR-based detectors. Co-DETR improves feature learning in the encoder and attention learning in the decoder through two main components: collaborative hybrid assignments training and customized positive queries generation. The collaborative hybrid assignments training scheme involves training multiple parallel auxiliary heads supervised by one-to-many label assignments such as ATSS and Faster RCNN which enhances the encoder's learning ability in end-to-end detectors. Additionally, customized positive queries are generated by extracting positive coordinates from these auxiliary heads which improves the efficiency of training positive samples in the decoder. During inference, these auxiliary heads are discarded introducing no additional parameters or computational cost to the original detector. Co-DETR also eliminates the need for handcrafted non-maximum suppression (NMS). The proposed approach is evaluated on various DETR variants including DAB-DETR, Deformable-DETR and DINO-Deformable-DETR with state of art results on COCO val dataset achieving an improvement from 58.5% to 59.5%. Moreover, when incorporated with ViT backbone it achieves impressive results of 66.0% AP on COCO test dev dataset and 67.9% AP on LVIS val dataset outperforming previous methods with significantly fewer model sizes. Overall, Co-DETR presents an effective solution for improving feature learning and attention learning in DETR based detectors while achieving state of art performance on various benchmark datasets.