# Establishing Best Practices for Building Rigorous Agentic Benchmarks

**Authors:** Yuxuan Zhu, Tengjun Jin, Yada Pruksachatkun, Andy Zhang, Shu Liu, Sasha Cui, Sayash Kapoor, Shayne Longpre, Kevin Meng, Rebecca Weiss, Fazl Barez, Rahul Gupta, Jwala Dhamala, Jacob Merizian, Mario Giulianelli, Harry Coppock, Cozmin Ududec, Jasjeet Sekhon, Jacob Steinhardt, Antony Kellerman  
**arXiv:** [2507.02825](https://arxiv.org/abs/2507.02825)  
**Date:** 2025-07-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2507.02825v1/

## Key points

- Importance of benchmarks in tracking progress in AI highlighted
- Need for agentic benchmarks to evaluate AI agents on complex real-world tasks emphasized
- Existing agentic benchmarks may have issues related to task setup or reward design, leading to under- or overestimation of agent performance
- Introduction of the Agentic Benchmark Checklist (ABC) to address challenges and ensure rigorous evaluation of AI agents
- ABC framework successfully reduces performance overestimation by 33% when applied to CVE-Bench
- Critical role of well-designed agentic benchmarks in accurately assessing AI agent performance underscored
- Emphasis on following established guidelines to avoid misleading evaluations

## Summary

In the paper titled "Establishing Best Practices for Building Rigorous Agentic Benchmarks," authored by a team including Yuxuan Zhu, Tengjun Jin, and others, the importance of benchmarks in tracking progress in AI is highlighted. As AI agents continue to advance in capabilities, the need for agentic benchmarks to evaluate these agents on complex real-world tasks becomes crucial. These benchmarks typically assess agent capabilities by measuring task outcomes through specific reward designs. However, the authors point out that many existing agentic benchmarks suffer from issues related to task setup or reward design. For instance, benchmarks like SWE-bench Verified may use insufficient test cases, while others like TAU-bench may count empty responses as successful outcomes. Such flaws can result in significant under- or overestimation of agent performance by up to 100% relative to actual abilities. To address these challenges and ensure rigorous evaluation of AI agents, the authors introduce the Agentic Benchmark Checklist (ABC). This checklist comprises guidelines derived from their own experience in benchmark-building, a survey of best practices in the field, and previously reported issues. When applied to CVE-Bench—a benchmark with a particularly complex evaluation design—the ABC framework successfully reduces performance overestimation by 33%. The research presented in this paper sheds light on the critical role of well-designed agentic benchmarks in accurately assessing AI agent performance and emphasizes the importance of following established guidelines to avoid misleading evaluations. The comprehensive approach taken by the authors in developing the ABC checklist serves as a valuable resource for researchers and practitioners striving for precision and reliability in benchmarking AI systems.