# FLAM: Frame-Wise Language-Audio Modeling

**Authors:** Yusong Wu, Christos Tsirigotis, Ke Chen, Cheng-Zhi Anna Huang, Aaron Courville, Oriol Nieto, Prem Seetharaman, Justin Salamon  
**arXiv:** [2505.05335](https://arxiv.org/abs/2505.05335)  
**Date:** 2025-05-08  
**Summary page:** https://www.summarizepaper.com/en/paper/2505.05335v1/

## Key points

- FLAM (Frame-Wise Language-Audio Modeling) is designed to address limitations of existing multi-modal audio-language models in frame-wise audio understanding.
- FLAM introduces an innovative approach for precise localization of specific sound events, overcoming challenges faced by traditional sound event detection models.
- The model employs a memory-efficient and calibrated frame-wise objective with logit adjustment to address spurious correlations during training.
- FLAM leverages a large-scale dataset with diverse audio events, LLM-generated captions, and simulation techniques for frame-wise supervision.
- Experimental results show that FLAM significantly enhances open-vocabulary localization capabilities while maintaining strong performance in global retrieval and downstream tasks.
- The model's approach to frame-wise audio-language alignment has the potential to drive innovation in the field of audio understanding through natural language queries.

## Summary

FLAM (Frame-Wise Language-Audio Modeling) is a cutting-edge model designed to address the limitations of existing multi-modal audio-language models in frame-wise audio understanding. This innovative approach opens up new possibilities for  audio understanding, benefiting applications such as content indexing, accessibility, and multimedia retrieval. While these models excel at text-audio retrieval, they struggle with pinpointing when specific sound events occur. Traditional sound event detection models are limited by pre-defined categories, making them ineffective for real-world scenarios with out-of-distribution events. In response to these challenges, FLAM introduces an  that can precisely localize specific sound events. By employing a memory-efficient and calibrated frame-wise objective with logit adjustment, FLAM effectively addresses spurious correlations such as event dependencies and label imbalances during training. To enable frame-wise supervision, the model leverages a large-scale dataset with diverse audio events, LLM-generated captions, and simulation techniques. Experimental results and case studies demonstrate that FLAM significantly enhances open-vocabulary localization capabilities while maintaining strong performance in global retrieval and downstream tasks. The model's innovative approach to frame-wise audio-language alignment has the potential to drive innovation in the field of , allowing for more accurate and precise understanding of complex audio data through natural language queries. While there are no significant ethical risks associated with FLAM, the researchers emphasize responsible use of the model in real-world scenarios. The project acknowledges the valuable contributions of individuals like Yuanbo Hou and Samuel Lavoie who provided insightful discussions and advice throughout its development. FLAM represents a significant advancement in multimodal learning and has the potential to drive innovation in the field of audio understanding through natural language queries.