# Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows

**Authors:** Yilun Yao, Xinyu Tan, Chao-Hsuan Liu, Yaoming Li, Zhengyang Wang, Wenhan Yu, Zhewen Tan, Yuxuan Tian, Guangxiang Zhao, Lin Sun, Xiangzheng Zhang, Tong Yang  
**arXiv:** [2605.27922](https://arxiv.org/abs/2605.27922)  
**Date:** 2026-05-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2605.27922v1/

## Key points

- Shift towards assessing executable agent benchmarks in Language Model (LLM) evaluation
- Existing benchmarks emphasize multi-step execution processes with external state considerations
- Importance of the harness in managing context, tools, state constraints, permissions, tracing mechanisms, and recovery processes
- Harness-Bench provides a diagnostic benchmark to evaluate configuration-level harness effects in agent workflows
- Benchmark consists of 106 sandboxed offline tasks constructed from practical agent-use patterns
- Significant variation observed in completion rates and process quality efficiency levels across different model-harness pairings
- Importance of reporting agent capability at the model-harness configuration level
- Identification of recurring execution-alignment failures where plausible reasoning becomes disconnected from tool feedback or workspace state
- Harness-Bench serves as a reproducible foundation for diagnosing and enhancing reliable, efficient, and auditable agent execution stacks

## Summary

In the field of Language Model (LLM) evaluation, there has been a shift towards assessing executable agent benchmarks rather than static language and reasoning benchmarks. This shift is evident in recent workflow-agent benchmarks like AgentBench, GAIA, and Claw-Eval which emphasize multi-step execution processes with external state considerations. However, existing benchmarks often overlook the importance of the harness - the system layer that manages context, tools, state constraints, permissions, tracing mechanisms, and recovery processes in agent workflows. <br/><br/>
Harness-Bench addresses this gap by providing a diagnostic benchmark specifically designed to evaluate configuration-level harness effects in realistic agent workflows. It evaluates different harness configurations across multiple model backends under shared task environments while preserving each harness's native execution behavior. The benchmark consists of 106 sandboxed offline tasks constructed from practical agent-use patterns and manually reviewed for realism and solvability. Each run records final artifacts, execution traces, usage statistics, and validator outputs to enable analysis beyond just final completion.<br/><br/>
Through 5,194 execution trajectories analyzed in Harness-Bench,<br/> significant variation is observed in completion rates,<br/> process quality efficiency levels across different model-harness pairings.<br/> These findings suggest that it is essential to report agent capability at the model-harness configuration level rather than solely attributing it to the base model.<br/><br/>
The analysis also identifies recurring execution-alignment failures where plausible reasoning becomes disconnected from tool feedback or workspace state.<br/><br/>
Harness-Bench serves as a reproducible foundation for diagnosing and enhancing reliable,<br/> efficient,<br/> and auditable agent execution stacks by providing controlled evaluations of harness effects across various representative harnesses and end-to-end workflows.