AutoHarness: improving LLM agents by automatically synthesizing a code harness

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Not relevant

   Self-Harness: Harnesses That Improve Themselves

   Hangfan Zhang, Shao Zhang, Kangcong Li et al. · 2026 · Computation and Language · preprint, not peer reviewed · cited 123 times · AI summary available

   Abstract

   The performance of LLM-based agents is jointly shaped by their base models and the harnesses that mediate their interaction with the environment. Because different models exhibit distinct behaviors, effective harness design is inherently model-specific. Yet agent harnesses are still largely engineered by human experts, a paradigm that scales poorly as modern LLMs become increasingly diverse and rapidly evolving. In this paper, we introduce Self-Harness, a new paradigm in which an LLM-based agent improves its own operating harness, without relying on human engineers or stronger external agents. We operationalize Self-Harness as an iterative loop with three stages: Weakness Mining, which ident…

2. Very close Not relevant

   Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses

   Jiahang Lin, Shichun Liu, Chengjun Pan et al. · 2026 · Computation and Language · preprint, not peer reviewed · cited 166 times · AI summary available

   Abstract

   Harnesses are now central to coding-agent performance, mediating how models interact with tools and execution environments. Yet harness engineering remains a manual craft, because automating it faces a heterogeneous action space across editable components, voluminous trajectories that bury actionable signal, and edits whose effect is hard to attribute. We introduce Agentic Harness Engineering (AHE), a closed loop that addresses these challenges through three matched observability pillars: (1) component observability gives every editable harness component a file-level representation so the action space is explicit and revertible; (2) experience observability distills millions of raw trajector…

3. Very close Cites this paper Not relevant

   Natural-Language Agent Harnesses

   Linyue Pan, Lexiao Zou, Shuo Guo et al. · 2026 · Computation and Language · preprint, not peer reviewed · cited 60 times · AI summary available

   Abstract

   Agent performance is strongly shaped by the surrounding harness: the external execution system around a model that organizes a task run. Yet this logic is usually buried in tightly coupled controller code, which makes harnesses hard to inspect, compare, transfer, and ablate. This paper asks whether the reusable design pattern of an agent harness can be represented as an executable natural-language object. We introduce Natural-Language Agent Harnesses (NLAHs), editable documents that describe run-level harness policy, and Intelligent Harness Runtime (IHR), a shared runtime that interprets these documents into agent calls, handoffs, state updates, validation gates, and artifact contracts. Acro…

4. Very close Cites this paper Not relevant

   JIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness Evolution

   Gui-Bin Zhang, Leo Lu, Fang-Zhou Xie et al. · 2026 · cited 17 times · Not summarized yet

   Abstract

   Agent capability is not determined by the model alone. The agent harness, encompassing memory management, planning strategy, action protocol, and tool/skill orchestration, can dominate the contribution of the underlying foundation model. Yet harness design remains manual, task-specific, and fundamentally unscalable. We present JIT-Agent, a harness intelligence model trained to synthesize task-adaptive agent harnesses on the fly for arbitrary off-the-shelf agentic LLMs. We formalize the agent harness as a composable, machine-generatable artifact governed by a fixed four-module protocol, and train JIT-Agent to customize harnesses for a given task at hand, repair harnesses for stable and reliab…

5. Close Not relevant

   Code as Agent Harness

   Xuying Ning, Katherine Tieu, Dongqi Fu et al. · 2026 · Computation and Language · preprint, not peer reviewed · cited 54 times · AI summary available

   Abstract

   Recent large language models (LLMs) have demonstrated strong capabilities in understanding and generating code, from competitive programming to repository-level software engineering. In emerging agentic systems, code is no longer only a target output. It increasingly serves as an operational substrate for agent reasoning, acting, environment modeling, and execution-based verification. We frame this shift through the lens of agent harnesses and introduce code as agent harness: a unified view that centers code as the basis for agent infrastructure. To systematically study this perspective, we organize the survey around three connected layers. First, we study the harness interface, where code c…

6. Close Cites this paper Not relevant

   MoMHa: Multi-Objective Optimization of LLM Harnesses over Accuracy, Safety, and Tokens

   Subhojyoti Mukherjee, Mahmud Tanjim · 2026 · cited 1 time · Not summarized yet

   Abstract

   Most work on improving large language models treats accuracy as the sole objective. We argue that the harness, the Python code surrounding the model that constructs prompts, routes calls, and parses outputs, is a first-class design surface whose quality is inherently multi-objective: an accurate harness that refuses no unsafe request, or that consumes an order of magnitude more tokens, is not a good harness. We present Meta-Harness, a system that casts harness design as search over three per-domain objectives (accuracy, behavioural safety, and token cost) solved by an agentic proposer (Claude Code) with full filesystem access to prior harness source, execution traces, and scoring artifacts. …

7. Close Not relevant

   Effective Harness Engineering for Algorithm Discovery with Coding Agents

   Yoichi Ishibashi, Taro Yano, Masafumi Oyamada · 2026 · Software Engineering · preprint, not peer reviewed · cited 8 times · AI summary available

   Abstract

   AlphaEvolve and FunSearch have demonstrated the potential of combining large language models (LLMs) with evolutionary search for automated algorithm discovery. However, discovery success is shaped not only by model capability but also significantly by the design of the execution infrastructure, i.e., the harness. This paper investigates effective harness design through three questions: under a fixed token budget, is it better to produce many algorithms with brief thought or fewer algorithms with deeper thought? How should the harness handle evaluation hacks, where generated programs exploit the scoring function? And how can agents that require full filesystem access execute safely in paralle…

8. Close Cites this paper Uses this paper in depth Not relevant

   LiteEvo: Automated, Cost-Efficient Harness Evolution for Generalization to Unseen Tasks

   Euntae Choi, Su-Min Song, Sungjoo Yoo · 2026 · cited 1 time · Not summarized yet

   Abstract

   An LLM agent is defined by two things: the weights inside its model and the harness of components assembled around it. Harnesses are still handcrafted, and HarnessX, which evolves them automatically, starts each benchmark from a handcrafted harness, reports gains on the tasks it evolved on, and budgets 100 to 175 million meta-agent tokens per benchmark. We propose LiteEvo, a lightweight harness-evolution algorithm whose tool-free meta-agents mine agent trajectories for reusable components, curate them into a versioned library, and compose each round's harness from it, starting every benchmark from the same neutral harness and never naming the benchmark. Evolving on the graded tasks of five a…

Prior work it builds on

1. Close Not relevant

   Eureka: Human-Level Reward Design via Coding Large Language Models

   Yecheng Jason Ma, William Liang, Guanzhi Wang et al. · 2023 · Robotics · cited 851 times · AI summary available

   Abstract

   Large Language Models (LLMs) have excelled as high-level semantic planners for sequential decision-making tasks. However, harnessing them to learn complex low-level manipulation tasks, such as dexterous pen spinning, remains an open problem. We bridge this fundamental gap and present Eureka, a human-level reward design algorithm powered by LLMs. Eureka exploits the remarkable zero-shot generation, code-writing, and in-context improvement capabilities of state-of-the-art LLMs, such as GPT-4, to perform evolutionary optimization over reward code. The resulting rewards can then be used to acquire complex skills via reinforcement learning. Without any task-specific prompting or pre-defined rewar…

2. Close Not relevant

   Code as Policies: Language Model Programs for Embodied Control

   Jacky Liang, Wen-Long Huang, F. Xia et al. · 2022 · IEEE International Conference on Robotics and Automation · cited 2,203 times · Not summarized yet

   Abstract

   Large language models (LLMs) trained on code-completion have been shown to be capable of synthesizing simple Python programs from docstrings [1]. We find that these code-writing LLMs can be re-purposed to write robot policy code, given natural language commands. Specifically, policy code can express functions or feedback loops that process perception outputs (e.g., from object detectors [2], [3]) and parameterize control primitive APIs. When provided as input several example language commands (formatted as comments) followed by corresponding policy code (via few-shot prompting), LLMs can take in new commands and autonomously re-compose API calls to generate new policy code respectively. By c…

3. Related Used in depth by this paper Not relevant

   Code Repair with LLMs gives an Exploration-Exploitation Tradeoff

   Hao Tang, Ke-Ya Hu, Jin Peng Zhou et al. · 2024 · Neural Information Processing Systems · cited 90 times · Not summarized yet

   Abstract

   Iteratively improving and repairing source code with large language models (LLMs), known as refinement, has emerged as a popular way of generating programs that would be too complex to construct in one shot. Given a bank of test cases, together with a candidate program, an LLM can improve that program by being prompted with failed test cases. But it remains an open question how to best iteratively refine code, with prior work employing simple greedy or breadth-first strategies. We show here that refinement exposes an explore-exploit tradeoff: exploit by refining the program that passes the most test cases, or explore by refining a lesser considered program. We frame this as an arm-acquiring …

4. Close Not relevant

   LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations

   Anian Ruoss, Fabio Pardo, Harris Chan et al. · 2024 · International Conference on Machine Learning · cited 40 times · Not summarized yet

   Abstract

   In this paper, we present a benchmark to pressure-test today's frontier models' multimodal decision-making capabilities in the very long-context regime (up to one million tokens) and investigate whether these models can learn from large numbers of expert demonstrations in their context. We evaluate the performance of Claude 3.5 Sonnet, Gemini 1.5 Flash, Gemini 1.5 Pro, Gemini 2.0 Flash Experimental, GPT-4o, o1-mini, o1-preview, and o1 as policies across a battery of simple interactive decision-making tasks: playing tic-tac-toe, chess, and Atari, navigating grid worlds, solving crosswords, and controlling a simulated cheetah. We study increasing amounts of expert demonstrations in the context…

5. Related Not relevant

   Code World Models for General Game Playing

   Wolfgang Lehrach, D. Hennes, Miguel Lázaro-Gredilla et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 31 times · Not summarized yet

   Abstract

   Large Language Models (LLMs) reasoning abilities are increasingly being applied to classical board and card games, but the dominant approach -- involving prompting for direct move generation -- has significant drawbacks. It relies on the model's implicit fragile pattern-matching capabilities, leading to frequent illegal moves and strategically shallow play. Here we introduce an alternative approach: We use the LLM to translate natural language rules and game trajectories into a formal, executable world model represented as Python code. This generated model -- comprising functions for state transition, legal move enumeration, and termination checks -- serves as a verifiable simulation engine …

Papers that cite it

1. Close Not relevant

   HarnessForge: Joint Harness and Policy Evolution for Adaptive Agent Systems

   Ming-Ju Chen, Can-Can Lv, Guibin Zhang et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 25 times · Not summarized yet

   Abstract

   LLM agents are increasingly expected to operate across heterogeneous task regimes that require distinct execution paradigms. This challenges fixed agent systems and motivates system-level meta-adaptation beyond isolated component updates. While existing works have adapted external harness or trained underlying reasoning policies, full-system adaptation remains insufficiently characterized. The adaptation space between structure and execution is rarely made explicit, and the compatibility between the external harness and the internal reasoner is not optimized jointly. We propose HarnessForge, a meta-adaptive framework for evolving LLM agent systems. HarnessForge formulates an agent system as …

2. Close Not relevant

   From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flaws

   Meng-Zhuo Chen, Junjie Wang, Zhe Liu et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 36 times · Not summarized yet

   Abstract

   LLM agents increasingly rely on agent harness: the runtime infrastructure around the base model that defines execution environments, tool interfaces, context, lifecycle orchestration, observability, verification, and governance. Existing self-improving agents and automatic harness evolution methods mainly improve agents through runtime supervision, prompt optimization, workflow search, or harness modification based on final outcomes. However, they often fail to diagnose where the responsible evidence lies in failed trajectories and which harness implementation mechanism causes the unreliable behavior, resulting in broad, indirect, or poorly scoped changes. This paper proposes HarnessFix, a t…

3. Close Not relevant

   Harness Evolution as Learning: Approximation, Generalization, and Optimization Limits of Self-Improving Personal Agents

   Ze-Yu Gan, Zi-Xuan Gong, Yong Liu · 2026 · cited 2 times · Not summarized yet

   Abstract

   As the capabilities of large language models (LLMs) continue to advance, increasing attention is turning to how to translate their abilities into useful behavior. Personal agents bring this question into everyday settings, where models are expected to serve individual users and continually adapt to their preferences. With the underlying model held fixed, such adaptation relies on harness engineering: designing and evolving the surrounding layer that manages context, memory, tools, and execution. Despite rapid progress, the factors governing effective harness evolution remain insufficiently understood. To narrow this gap, we investigate three central questions concerning harness architecture,…

4. Close Not relevant

   Harness-R1: Learning to Edit Executable Runtime Harnesses from Agent Failure Trajectories

   Shuai Shao, Kang-Ning Zhang, Qingyao Li et al. · 2026 · cited 19 times · Not summarized yet

   Abstract

   Agents built around large language models continually accumulate interaction trajectories during deployment, yet their behavior typically remains fixed. Beyond updating model weights, these trajectories can improve the agent harness that constructs context, mediates tools, validates actions, and recovers execution. We introduce Harness-R1, the first method, to our knowledge, that makes failure-conditioned, lifecycle-wide editing of an existing executable runtime a learned capability. It post-trains a dedicated harness engineer with online reinforcement learning so that its edits are optimized for the realized task success they produce, rather than proposed by a fixed editor. A separate 9B en…

5. Close Not relevant

   SafeHarness: Lifecycle-Integrated Security Architecture for LLM-based Agent Deployment

   Xi-Xun Lin, Yang Liu, Yan-Cheng Chen et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 16 times · Not summarized yet

   Abstract

   The performance of large language model (LLM) agents depends critically on the execution harness, the system layer that orchestrates tool use, context management, and state persistence. Yet this same architectural centrality makes the harness a high-value attack surface: a single compromise at the harness level can cascade through the entire execution pipeline. We observe that existing security approaches suffer from structural mismatch, leaving them blind to harness-internal state and unable to coordinate across the different phases of agent operation. In this paper, we introduce \safeharness{}, a security architecture in which four proposed defense layers are woven directly into the agent …