# LiveMCP-101: Stress Testing and Diagnosing MCP-enabled Agents on Challenging Queries

**Authors:** Ming Yin, Dinghan Shen, Silei Xu, Jianbing Han, Sixun Dong, Mian Zhang, Yebowen Hu, Shujian Liu, Simin Ma, Song Wang, Sathish Reddy Indurthi, Xun Wang, Yiran Chen, Kaiqiang Song  
**arXiv:** [2508.15760](https://arxiv.org/abs/2508.15760)  
**Date:** 2025-08-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2508.15760v1/

## Key points

- The Model Context Protocol (MCP) provides a standardized framework for integrating tools in AI agent tasks.
- LiveMCP-101 is a benchmark with 101 real-world queries that require multiple MCP tools for completion.
- Evaluation of AI agents using LiveMCP-101 shows challenges in orchestrating tools effectively, with success rates below 60%.
- Detailed analysis reveals failure modes and inefficiencies in token usage within current models, offering insights for improvement.
- The benchmark sets a rigorous standard for assessing real-world agent capabilities and aims to pave the way for autonomous AI systems proficient in executing complex tasks.
- Additional complexities arise from scenarios like market research projects and personal requests, highlighting the need for advanced AI systems capable of navigating diverse tool chains efficiently.
- Evaluation metrics include task success rate (TSR), average result score (ARS), average trajectory score (ATS), average tool calls, and average tokens used to provide comprehensive insights into model performance across different tasks.
- An innovative LLM-as-a-Judge approach has been employed to assess final outputs and execution trajectories, with human-expert studies conducted to compare judgments between experts and LLM judges.

## Summary

The use of AI agents in complex tasks has become increasingly important, with tool calling being a crucial aspect for interacting with the real world. To address this challenge, the Model Context Protocol (MCP) offers a standardized framework for integrating tools. However, there is still a gap in evaluating how effectively AI agents can solve multi-step tasks using diverse MCP tools in dynamic scenarios. To bridge this gap, LiveMCP-101 has been introduced as a benchmark consisting of 101 carefully curated real-world queries that require the coordinated use of multiple MCP tools such as web search, file operations, mathematical reasoning, and data analysis. This benchmark was refined through iterative LLM rewriting and manual review processes to ensure its accuracy and relevance. A novel evaluation approach has also been implemented for LiveMCP-101 by leveraging ground-truth execution plans instead of raw API outputs to better capture the evolving nature of real-world environments. Experimental results have shown that even cutting-edge LLMs struggle to achieve success rates above 60%, highlighting the significant challenges in orchestrating tools effectively. Detailed ablations and error analyses have further highlighted distinct failure modes and inefficiencies in token usage within current models, providing concrete insights for enhancing their performance. LiveMCP-101 sets a rigorous standard for assessing real-world agent capabilities and paves the way towards autonomous AI systems capable of reliably executing complex tasks through adept tool utilization. Furthermore, within the context of BrightPath Analytics' market research project on digital art engagement commissioned by artist Lucia Moretti and a personal request for NBA game tickets from a father to surprise his son, additional complexities arise that demand multi-step composition of heterogeneous MCP tools with precise parameterization and output handling. These scenarios underscore the need for advanced AI systems that can seamlessly navigate diverse tool chains to fulfill intricate user requests across various domains. The evaluation framework encompasses metrics such as task success rate (TSR), average result score (ARS), average trajectory score (ATS), average tool calls, and average tokens used to provide comprehensive insights into model performance across different tasks. Additionally, an innovative LLM-as-a-Judge approach has been employed to assess both final outputs and execution trajectories, with human-expert studies conducted to compare judgments between experts and LLM judges. Overall, this expanded summary showcases the depth and complexity of evaluating AI agent capabilities in real-world scenarios while emphasizing the importance of efficient tool orchestration for achieving successful task completion.