# Why Do Multi-Agent LLM Systems Fail?

**Authors:** Mert Cemri, Melissa Z. Pan, Shuyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, Matei Zaharia, Joseph E. Gonzalez, Ion Stoica  
**arXiv:** [2503.13657](https://arxiv.org/abs/2503.13657)  
**Date:** 2025-03-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2503.13657v1/

## Key points

- Authors address challenges hindering effectiveness of Multi-Agent Systems (MAS) compared to single-agent frameworks
- Conducted comprehensive study analyzing five prominent MAS frameworks across 150+ tasks with expert human annotators
- Identified 14 distinct failure modes and developed a comprehensive taxonomy applicable to various MAS frameworks
- Categorized failure modes into three groups: specification and system design failures, inter-agent misalignment, task verification and termination
- Integrated MASFT with LLM-as-a-Judge for scalable evaluation and explored interventions to prevent failures
- Proposed improved specification of agent roles and enhanced orchestration strategies as possible solutions
- Findings provide roadmap for future research in improving MAS performance
- Dataset and LLM annotator made open-source for further exploration and advancement in understanding challenges faced by Multi-Agent Systems

## Summary

By Mert Cemri et al., the authors address the challenges hindering the effectiveness of Multi-Agent Systems (MAS) in comparison to single-agent frameworks. Despite the increasing interest in MAS, their performance improvements on popular benchmarks are limited. To investigate this issue, the authors conducted a comprehensive study analyzing five prominent MAS frameworks across more than 150 tasks with the assistance of six expert human annotators. Through this analysis, they identified 14 distinct failure modes and developed a comprehensive taxonomy that can be applied to various MAS frameworks. This taxonomy was refined through agreements among three expert annotators per study, achieving a high Cohen's Kappa score of 0.88. The identified failure modes were categorized into three groups: specification and system design failures, inter-agent misalignment, and task verification and termination. To facilitate scalable evaluation, the authors integrated MASFT with LLM-as-a-Judge and explored potential interventions to prevent these failures. They proposed improved specification of agent roles and enhanced orchestration strategies as possible solutions but found that addressing these failures may require more complex approaches. The findings from this study provide a clear roadmap for future research in improving MAS performance. Furthermore, the authors have made their dataset and LLM annotator open-source, enabling further exploration and advancement in understanding and addressing challenges faced by Multi-Agent Systems. The collaborative effort of the authors sheds light on crucial aspects that need to be considered for enhancing the effectiveness of MAS frameworks in various applications.