# Co-attention Propagation Network for Zero-Shot Video Object Segmentation

**Authors:** Gensheng Pei, Yazhou Yao, Fumin Shen, Dan Huang, Xingguo Huang, Heng-Tao Shen  
**arXiv:** [2304.03910](https://arxiv.org/abs/2304.03910)  
**Date:** 2023-04-08  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.03910v1/

## Key points

- Zero-shot video object segmentation (ZS-VOS) aims to segment foreground objects in a video sequence without prior knowledge of these objects.
- Existing ZS-VOS methods struggle with distinguishing between foreground and background or keeping track of the foreground in complex scenarios.
- Researchers have proposed an encoder-decoder based hierarchical co-attention propagation network (HCPN) capable of tracking and segmenting objects.
- The HCPN model is built upon multiple collaborative evolutions of the parallel co-attention module (PCM) and the cross co-attention module (CCM).
- The method is progressively trained to achieve hierarchical spatio temporal feature propagation across the entire video.
- The HCPN model captures both appearance and motion features, addressing overreliance on optical flow estimation.
- Experimental results demonstrate that HCPN outperforms all previous methods on public benchmarks for ZS VOS tasks.
- This research provides a promising approach towards improving ZS VOS performance in complex scenarios where distinguishing between foreground and background can be challenging.

## Summary

The field of zero-shot video object segmentation (ZS-VOS) aims to segment foreground objects in a video sequence without prior knowledge of these objects. However, existing ZS-VOS methods often struggle to distinguish between foreground and background or to keep track of the foreground in complex scenarios. To address these challenges, a team of researchers led by Gensheng Pei, Yazhou Yao, Fumin Shen, Dan Huang, Xingguo Huang, and Heng-Tao Shen have proposed an encoder-decoder based hierarchical co-attention propagation network (HCPN) capable of tracking and segmenting objects. The proposed model is built upon multiple collaborative evolutions of the parallel co-attention module (PCM) and the cross co-attention module (CCM). PCM captures common foreground regions among adjacent appearance and motion features while CCM further exploits and fuses cross modal motion features returned by PCM. The method is progressively trained to achieve hierarchical spatio temporal feature propagation across the entire video. One common practice in ZS VOS is introducing motion information such as optical flow. However, this can lead to overreliance on optical flow estimation. The HCPN model addresses this issue by using collaborative evolutions of PCM and CCM modules that capture both appearance and motion features. Experimental results demonstrate that HCPN outperforms all previous methods on public benchmarks for ZS VOS tasks. This showcases its effectiveness for real world applications where prior knowledge about objects is not available. Overall, this research provides a promising approach towards improving ZS VOS performance in complex scenarios where distinguishing between foreground and background can be challenging.