# Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics

**Authors:** Siddhant Arora, Zhiyun Lu, Chung-Cheng Chiu, Ruoming Pang, Shinji Watanabe  
**arXiv:** [2503.01174](https://arxiv.org/abs/2503.01174)  
**Date:** 2025-03-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2503.01174v1/

## Key points

- Authors Siddhant Arora, Zhiyun Lu, Chung-Cheng Chiu, Ruoming Pang, and Shinji Watanabe explore the potential of audio foundation models (FMs) in enhancing conversational modeling.
- The study addresses the lack of comprehensive evaluation of FMs in facilitating natural and interactive conversations.
- Importance of FMs engaging in fluent turn-taking without speech overlap or prolonged silence for meaningful interactions.
- Introduction of a novel evaluation protocol involving a supervised model to assess turn-taking proficiency in spoken dialog systems.
- User study reveals issues such as failure to discern speaking cues, aggressive interruptions, and lack of backchanneling in existing systems.
- Evaluation extends to multiple open-source and proprietary audio FMs sourced from Switchboard to measure comprehension and forecasting of turn-taking events.
- Plan to release the evaluation platform as an open-source resource to advance conversational AI systems.

## Summary

In their paper titled "Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics," authors Siddhant Arora, Zhiyun Lu, Chung-Cheng Chiu, Ruoming Pang, and Shinji Watanabe explore the potential of audio foundation models (FMs) in enhancing conversational modeling. The study addresses the lack of comprehensive evaluation of these FMs in their ability to facilitate natural and interactive conversations. The researchers emphasize the importance of FMs being able to engage in fluent turn-taking without excessive speech overlap or prolonged periods of silence to ensure meaningful interactions with users. To assess the capability of recently proposed audio FMs in understanding, predicting, and executing turn-taking events, the authors introduce a novel evaluation protocol. This protocol involves utilizing a supervised model trained to predict turn-taking events in human-human conversations as a judge to evaluate spoken dialog systems' turn-taking proficiency. Through this approach, the researchers conduct a thorough user study that sheds light on existing spoken dialogue systems' performance in turn-taking scenarios. They uncover insights such as instances where systems fail to discern appropriate speaking cues, exhibit overly aggressive interruptions, and lack adequate backchanneling. Furthermore, the study extends its evaluation to multiple open-source and proprietary audio FMs accessible via APIs by subjecting them to meticulously curated test benchmarks sourced from Switchboard. The goal is to measure these models' capacity to comprehend and forecast turn-taking events while identifying areas for enhancement. Ultimately, the researchers plan to release their evaluation platform as an open-source resource to foster advancements in conversational AI systems. Accepted at ICLR 2025, this research contributes valuable insights into the evolving landscape of audio foundation models and their potential impact on improving conversational interactions through enhanced turn-taking dynamics.