# Bootstrapping Audio-Visual Segmentation by Strengthening Audio Cues

**Authors:** Tianxiang Chen, Zhentao Tan, Tao Gong, Qi Chu, Yue Wu, Bin Liu, Le Lu, Jieping Ye, Nenghai Yu  
**arXiv:** [2402.02327](https://arxiv.org/abs/2402.02327)  
**Date:** 2024-02-04  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.02327v2/

## Key points

- Integration of audio and visual cues in multi-modality research is a significant area of interest
- Introduction of the audio-visual segmentation (AVS) task for identifying and segmenting auditory objects within video frames using audio guidance
- Modality imbalance issue in existing AVS methods where visual features overshadow those from the audio domain
- Proposal of AVSAC approach with Bidirectional Audio-Visual Decoder (BAVD) to enhance bidirectional integration of audio cues and promote continuous interaction between auditory and visual modalities
- Strategy for achieving audio-visual frame-wise synchrony to amplify auditory components within visual features for balanced learning process
- AVSAC surpasses previous benchmarks in AVS performance, demonstrating efficacy in addressing modality imbalances
- Collaboration by authors Tianxiang Chen, Zhentao Tan, Tao Gong, Qi Chu, Yue Wu, Bin Liu, Le Lu, Jieping Ye, and Nenghai Yu has led to a groundbreaking advancement in enhancing audio-visual segmentation through strengthened audio cues.

## Summary

, , , , 

In the realm of multi-modality research, the integration of audio and visual cues has become a topic of significant interest. A recent development in this field is the introduction of the audio-visual segmentation (AVS) task, which seeks to identify and segment auditory objects within video frames using audio guidance. However, existing AVS methods have been plagued by a modality imbalance where visual features tend to overshadow those from the audio domain. This imbalance arises from a lack of bidirectional integration of audio cues, leading to skewed feature representations that hinder the learning of cohesive audio-visual representations and may result in segmentation inaccuracies. To tackle this issue head-on, a novel approach known as AVSAC has been proposed. At its core lies the Bidirectional Audio-Visual Decoder (BAVD), equipped with integrated bidirectional bridges that enhance audio cues and promote continuous interaction between the auditory and visual modalities. By fostering bidirectional communication between these modalities, AVSAC effectively reduces the modality imbalance, thereby facilitating more efficient learning of integrated audio-visual representations. Additionally, a strategy for achieving audio-visual frame-wise synchrony has been introduced as fine-grained guidance for BAVD. This strategy works to amplify the presence of auditory components within visual features, ultimately contributing to a more balanced learning process for audio-visual representation. Extensive experimentation has demonstrated that AVSAC surpasses previous benchmarks in AVS performance, showcasing its efficacy in addressing challenges posed by modality imbalances in current AVS methodologies. The collaborative efforts of authors Tianxiang Chen, Zhentao Tan, Tao Gong, Qi Chu, Yue Wu, Bin Liu, Le Lu, Jieping Ye, and Nenghai Yu have culminated in a groundbreaking advancement towards enhancing audio-visual segmentation through strengthened audio cues.