# Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Survey

**Authors:** Chih-Kai Yang, Neo S. Ho, Hung-yi Lee  
**arXiv:** [2505.15957](https://arxiv.org/abs/2505.15957)  
**Date:** 2025-05-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2505.15957v1/

## Key points

- Large audio-language models (LALMs) have advanced traditional large language models (LLMs) by incorporating auditory processing.
- LALMs excel in auditory tasks such as speech recognition, audio generation, and multimodal reasoning.
- A comprehensive survey proposed a taxonomy for evaluating LALMs across four dimensions: General Auditory Awareness and Processing, Knowledge and Reasoning, Dialogue-oriented Ability, and Fairness, Safety, and Trustworthiness.
- The taxonomy aims to provide clear guidelines for researchers and practitioners in the field to evaluate LALMs systematically.
- Categorizing evaluations based on objectives offers a holistic view of LALM capabilities and identifies areas for improvement and future research directions.

## Summary

In recent years, the field of large audio-language models (LALMs) has seen significant advancements. These LALMs have expanded the capabilities of traditional large language models (LLMs) by incorporating auditory processing. They are designed to excel in a wide range of auditory tasks and have shown promise in various domains such as speech recognition, audio generation, and multimodal reasoning. Despite the emergence of benchmarks to evaluate LALM performance, there is a lack of a structured taxonomy to categorize and assess these models comprehensively. To address this gap, a comprehensive survey was conducted to propose a systematic taxonomy for evaluating LALMs across four key dimensions: General Auditory Awareness and Processing, Knowledge and Reasoning, Dialogue-oriented Ability, and Fairness, Safety, and Trustworthiness. Each dimension provides detailed insights into the specific objectives and challenges associated with evaluating LALMs in those areas. This survey represents the first focused effort on systematically evaluating LALMs and aims to provide clear guidelines for researchers and practitioners in the field. By categorizing evaluations based on their objectives, this taxonomy offers a holistic view of LALM capabilities and highlights areas for improvement and future research directions. The surveyed papers will be made available to support ongoing advancements in the field, ensuring that LALMs continue to evolve towards universal proficiency across auditory tasks.