# A Closer Look at Weakly-Supervised Audio-Visual Source Localization

**Authors:** Shentong Mo, Pedro Morgado  
**arXiv:** [2209.09634](https://arxiv.org/abs/2209.09634)  
**Date:** 2022-08-30  
**Summary page:** https://www.summarizepaper.com/en/paper/2209.09634v1/

## Key points

- Authors Shentong Mo and Pedro Morgado focus on predicting the location of visual sound sources in videos.
- Traditional ground-truth annotation methods for sounding objects are costly, leading to the development of weakly-supervised localization methods.
- Existing evaluation protocols have flaws, such as early stopping with fully annotated datasets and assuming sound sources are always present.
- The authors propose an extension to benchmarks like Flickr SoundNet and VGG-Sound Sources by including negative samples in the test set.
- New metrics are introduced to balance localization accuracy and recall for a more comprehensive evaluation of prior methods.
- Many existing approaches struggle to identify negatives and suffer from overfitting due to heavy reliance on early stopping.
- Mo and Morgado present a novel approach using extreme visual dropout and momentum encoders, achieving state-of-the-art performance on benchmarks.
- The authors provide their code and pre-trained models for further research on GitHub (https://github.com/stoneMo/SLAVC).
- This study emphasizes the importance of refining evaluation protocols in weakly-supervised audio-visual source localization research.

## Summary

In their paper titled "A Closer Look at Weakly-Supervised Audio-Visual Source Localization," authors Shentong Mo and Pedro Morgado delve into the challenging task of predicting the location of visual sound sources in videos. The traditional approach of collecting ground-truth annotations for sounding objects is costly, leading to the development of weakly-supervised localization methods that leverage the co-occurrence of audio and visual signals in datasets without bounding-box annotations. However, existing evaluation protocols have significant flaws, allowing early stopping with fully annotated datasets and assuming sound sources are present at all times. To address these issues, the authors propose an extension to popular benchmarks like Flickr SoundNet and VGG-Sound Sources by including negative samples in the test set. They introduce new metrics that balance localization accuracy and recall, enabling a more comprehensive evaluation of prior methods. Through this extended protocol, they discover that many existing approaches struggle to identify negatives and suffer from overfitting due to heavy reliance on early stopping. In response to these findings, Mo and Morgado present a novel approach for visual sound source localization that tackles both problems effectively. By incorporating extreme visual dropout and momentum encoders, their method combats overfitting and achieves state-of-the-art performance on Flickr SoundNet and VGG-Sound Source benchmarks. The authors make their code and pre-trained models available for further research on GitHub (https://github.com/stoneMo/SLAVC). This study sheds light on the importance of refining evaluation protocols in weakly-supervised audio-visual source localization research and offers a promising solution to enhance model performance in this domain.