# Agent-as-a-Judge: Evaluate Agents with Agents

**Authors:** Mingchen Zhuge, Changsheng Zhao, Dylan Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, Yangyang Shi, Vikas Chandra, Jürgen Schmidhuber  
**arXiv:** [2410.10934](https://arxiv.org/abs/2410.10934)  
**Date:** 2024-10-14  
**Summary page:** https://www.summarizepaper.com/en/paper/2410.10934v1/

## Key points

- Researchers introduce the Agent-as-a-Judge framework to address inadequacy of current evaluation techniques for agentic systems
- The DevAI dataset is released, containing 55 AI development tasks with detailed requirements and preferences
- Benchmarking of top open-source code generation agentic frameworks using the DevAI dataset shows superiority of Agent-as-a-Judge framework over LLM-as-a-Judge method
- Agent-as-a-Judge framework performs comparably to human evaluators in proof-of-concept test by providing rich and reliable reward signals for self-improvement
- Motivation behind creating DevAI benchmark is to provide realistic benchmarks for automated AI development tasks that reflect entire process and offer sufficient reward signals
- Human evaluations show promising results of Agent-as-a-Judge framework in evaluating agentic systems' performance, reducing manual oversight and training times

## Summary

In this study, the researchers address the inadequacy of contemporary evaluation techniques for agentic systems by introducing the Agent-as-a-Judge framework. This innovative approach allows agentic systems to evaluate other agentic systems, providing a more comprehensive and dynamic evaluation process compared to traditional methods. To demonstrate the effectiveness of this approach, the researchers release the DevAI dataset which consists of 55 AI development tasks with detailed hierarchical requirements and preferences. By benchmarking three top open-source code generation agentic frameworks using the DevAI dataset, they offer a more thorough analysis than previous evaluations. The results show that the Agent-as-a-Judge framework outperforms the LLM-as-a-Judge method and performs comparably to human evaluators in a proof-of-concept test. This is achieved by incorporating agentic features that enable intermediate feedback throughout task-solving processes, providing rich and reliable reward signals necessary for self-improvement in modern agentic systems. The motivation behind creating the DevAI benchmark stems from a lack of realistic benchmarks for automated AI development tasks. Existing benchmarks often focus on final outcomes rather than reflecting the entire development process or providing sufficient reward signals for long-horizon tasks. The researchers emphasize practical software scenarios where tasks are complex and require human or agentic assistance. Through human evaluations conducted on baseline executions, it is evident that the Agent-as-a-Judge framework offers promising results in evaluating agentic systems' performance. The comparison with human evaluators highlights the potential of this new approach in providing valuable insights into AI development tasks while reducing manual oversight and training times. Overall, this study marks a significant advancement in evaluating modern agentic systems effectively and efficiently through innovative frameworks like Agent-as-a-Judge.