# AV-SAM: Segment Anything Model Meets Audio-Visual Localization and Segmentation

**Authors:** Shentong Mo, Yapeng Tian  
**arXiv:** [2305.01836](https://arxiv.org/abs/2305.01836)  
**Date:** 2023-05-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.01836v1/

## Key points

- The Segment Anything Model (SAM) is effective in visual segmentation tasks.
- AV-SAM is a framework that combines audio and visual information for audio-visual tasks.
- AV-SAM leverages pixel-wise audio-visual fusion by aggregating cross-modal representations from audio and visual features.
- The aggregated cross-modal features are used to generate final audio-visual segmentation masks.
- AV-SAM achieves competitive performance in sounding object localization and segmentation.
- Extensive experiments were conducted on the Flickr-SoundNet and AVSBench datasets.
- AV-SAM provides accurate sounding object masks corresponding to the audio input.

## Summary

The Segment Anything Model (SAM) has proven to be highly effective in visual segmentation tasks. To address the limited exploration of SAM's performance in audio-visual tasks such as visual sound localization and segmentation, the authors propose a framework called AV-SAM that combines audio and visual information to generate sounding object masks corresponding to the audio. AV-SAM leverages pixel-wise audio-visual fusion by aggregating cross-modal representations from audio features and visual features obtained from the pre-trained image encoder in SAM. These aggregated cross-modal features are then fed into the prompt encoder and mask decoder to generate final audio-visual segmentation masks. To evaluate the performance of AV-SAM, extensive experiments are conducted on two datasets: Flickr-SoundNet and AVSBench. The results demonstrate that AV-SAM achieves competitive performance in sounding object localization and segmentation. Overall, this work presents a simple yet effective approach for audio-visual localization and segmentation based on the Segment Anything Model. By leveraging cross-modal fusion of audio and visual features, AV-SAM provides accurate sounding object masks corresponding to the audio input.