# Tracking Anything with Decoupled Video Segmentation

**Authors:** Ho Kei Cheng, Seoung Wug Oh, Brian Price, Alexander Schwing, Joon-Young Lee  
**arXiv:** [2309.03903](https://arxiv.org/abs/2309.03903)  
**Date:** 2023-09-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.03903v1/

## Key points

- Challenges of annotating training data for video segmentation hinder the extension of end-to-end algorithms to new tasks
- Proposed solution: DEVA, a decoupled video segmentation approach
- DEVA consists of task-specific image-level segmentation and class/task-agnostic bi-directional temporal propagation
- Bi-directional propagation allows for (semi-)online fusion of segmentation hypotheses from different frames, resulting in coherent segmentation
- DEVA outperforms end-to-end approaches in data scarce tasks such as large vocabulary video panoptic segmentation, open world video segmentation, referring video segmentation, and unsupervised video object segmentation
- Comprehensive overview of related works in end-to-end video segmentation provided
- Bi-directional temporal propagation model denoises image segmentations and merges them with temporally propagated segmentations seamlessly
- Effectiveness of the model demonstrated through empirical evaluations on datasets including YouTube VOS and Cityscape VPS
- DEVA framework offers a promising solution to challenges associated with training data annotation for video segmentation by leveraging external data and incorporating existing universal image segmentation models

## Summary

The existing summary discusses the challenges of annotating training data for video segmentation and how it hinders the extension of end-to-end algorithms to new video segmentation tasks, particularly in large-vocabulary settings. To address this issue, the authors propose a decoupled video segmentation approach called DEVA. DEVA consists of task-specific image-level segmentation and class/task-agnostic bi-directional temporal propagation. This design allows for the use of an image-level model for the target task which is cheaper to train and a universal temporal propagation model that can be trained once and applied across tasks. To effectively combine these two modules, the authors utilize bi-directional propagation for (semi-)online fusion of segmentation hypotheses from different frames resulting in a coherent segmentation. The authors demonstrate that this decoupled formulation outperforms end-to-end approaches in various data scarce tasks such as large vocabulary video panoptic segmentation, open world video segmentation, referring video segmentation and unsupervised video object segmentation. In addition to introducing DEVA, the authors provide a comprehensive overview of related works in end to end video segmentation. They highlight recent advancements in this field and discuss how their proposed approach differs from existing methods. The authors also present their bi directional temporal propagation model in detail explaining how it denoises image segmentations and seamlessly merges them with temporally propagated segmentations. The effectiveness of this model is demonstrated through empirical evaluations on different datasets including YouTube VOS [69] and Cityscape VPS [27]. Overall, the proposed DEVA framework offers a promising solution to the challenges associated with training data annotation for video segmentation by leveraging external data and incorporating existing universal image segmentation models while achieving favorable results in various important video segmentation tasks.