Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses

Classés selon la proximité de leur titre et de leur résumé avec ceux de cet article. Sélectionnez-en pour les lire ensemble.

Les articles les plus proches

1. Très proche Cite cet article Pas pertinent

   HarnessX: A Composable, Adaptive, and Evolvable Agent Harness Foundry

   Ting-Yang Chen, Shuo Lu, Kangjia Zhao et al. · 2026 · arXiv.org · preprint, non relu par les pairs · cité 60 fois · Pas encore résumé

   Résumé scientifique

   AI agent performance depends critically on the runtime harness, comprising the prompts, tools, memory, and control flow that mediate how a model observes, reasons, and acts. Yet today's harnesses remain largely hand-crafted and static: each new model or task still demands bespoke scaffolding, and the rich traces produced during execution are rarely distilled back into systematic improvement. We introduce HarnessX, a foundry for composable, adaptive, and evolvable agent harnesses. HarnessX assembles typed harness primitives via a substitution algebra, adapts them through AEGIS, a trace-driven multi-agent evolution engine grounded in an operational mirror between symbolic adaptation and reinfo…

2. Très proche Cite cet article Pas pertinent

   HarnessCompass: Guiding Automatic Harness Evolution toward Generalizable and Effective Agent Harnesses

   Luan Zhang, Ruo-Chen Zhou, Dan-Dan Song et al. · 2026 · cité 24 fois · Pas encore résumé

   Résumé scientifique

   Harness design plays a critical role in agent performance by shaping how large language models (LLMs) perceive, reason over, and act within executable environments. Recent work has proposed automatic harness evolution, which iteratively improves the harness from agent--environment interactions. However, existing methods often overfit to the evolution tasks, rely exclusively on trajectory-derived signals, and optimize harness components jointly, causing interference across components. We propose HarnessCompass, a novel automatic harness evolution framework built around constrained evolution, proactive feedback, and component-wise optimization. HarnessCompass first enforces global constraints …

3. Très proche Cite cet article Utilise cet article en profondeur Pas pertinent

   Self-Evolving Harness on Multiple Tasks with the Agent as Its Own Optimizer

   Qian-Kai Xu · 2026 · cité 1 fois · Pas encore résumé

   Résumé scientifique

   A harness is the code around a language-model agent that organizes prompts, calls tools, manages context, and controls execution. As models grow stronger, recent work has begun to let agents improve their own harnesses, a line of work known as self-evolving harnesses. In most existing methods, a separate proposer running on a human-designed harness modifies the solver's harness, and a separate harness is evolved for each benchmark. Real-world tasks come from many domains, so both the evolution and the evaluation of a harness should cover a diverse range of tasks. We propose a framework close to recursive self-improvement: the same frozen model, on the same version of the harness, first solve…

4. Très proche Pas pertinent

   Adaptive Auto-Harness: Sustained Self-Improvement for Agentic System Deployment on Open-Ended Task Streams

   Ze-Wen Liu, Zhan Shi, Yi-Si Sang et al. · 2026 · arXiv.org · preprint, non relu par les pairs · cité 33 fois · Pas encore résumé

   Résumé scientifique

   Auto-harness systems such as A-Evolve, GEPA, and Meta-Harness improve LLM agents by optimizing prompts, skills, tools, memories, and supporting infrastructure from execution feedback, but they are typically evaluated on fixed offline benchmarks. Real deployments instead present open-ended task streams: histories grow without a fixed endpoint, heterogeneous tasks require different harnesses, and problem distributions shift over time. These challenges make a single repeatedly and densely updated harness brittle, causing performance degradation as accuracy peaks early and then declines. This motivates sustained harness construction with task-wise adaptation. We introduce Adaptive Auto-Harness, …

5. Très proche Cite cet article Pas pertinent

   Self-Harness: Harnesses That Improve Themselves

   Hangfan Zhang, Shao Zhang, Kangcong Li et al. · 2026 · Computation and Language · preprint, non relu par les pairs · cité 123 fois · Résumé IA disponible

   Résumé scientifique

   The performance of LLM-based agents is jointly shaped by their base models and the harnesses that mediate their interaction with the environment. Because different models exhibit distinct behaviors, effective harness design is inherently model-specific. Yet agent harnesses are still largely engineered by human experts, a paradigm that scales poorly as modern LLMs become increasingly diverse and rapidly evolving. In this paper, we introduce Self-Harness, a new paradigm in which an LLM-based agent improves its own operating harness, without relying on human engineers or stronger external agents. We operationalize Self-Harness as an iterative loop with three stages: Weakness Mining, which ident…

6. Très proche Cite cet article Pas pertinent

   Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable

   Ru-Han Wang, Yu-Cheng Shi, Zong-Xia Li et al. · 2026 · arXiv.org · preprint, non relu par les pairs · cité 15 fois · Pas encore résumé

   Résumé scientifique

   The capability of a modern AI agent depends not only on its foundation model but also on its harness, which constructs prompts, manages state, invokes tools, and coordinates execution. As models, APIs, environments, and requirements evolve, the harness must be continually modified. Before such a change can be made, a developer or coding agent must identify all code locations that implement the target behavior. This is difficult because production harnesses are large, tightly coupled, and behaviorally distributed, while modification requests describe what the system should do and repositories are organized by files and modules. Code search, repository indexing, and long-context processing eas…

7. Très proche Cite cet article Pas pertinent

   HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

   Yu-Hao Wu, Jing-Yuan Zhang, Jia-Jun Shi et al. · 2026 · cité 15 fois · Pas encore résumé

   Résumé scientifique

   As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure, commonly termed the agent harness. Changing this harness while holding model weights fixed can substantially alter task performance. Current agent evaluations typically report downstream performance under a chosen harness, leaving a model's ability to develop the harness itself comparatively underexplored. We introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to runnable infrastructure. HarnessDev covers two stages. In Creation, the agent starts from a minimal seed and a small number of cases, then builds a complete e…

8. Très proche Cite cet article Utilise cet article en profondeur Pas pertinent

   Causal Improvement Graph for Agentic Harness Optimization

   Jun-Jie Zhang, Shunyu Liu, Hao-Yu Wang et al. · 2026 · Pas encore résumé

   Résumé scientifique

   Agentic Harness is the runtime that constructs task context and controls execution flow, thereby shaping overall agent performance. Given a fixed model and external evaluation, automated Harness optimization seeks to improve this runtime through an iterative proposal--evaluation loop to better solve target tasks. Existing meta-harness methods mainly adopt proposer-centric discovery, in which an LLM-based proposer integrates accumulated experimental findings to determine subsequent Harness revisions. This places the burden of maintaining the evolving improvement state on the proposer as history expands and its underlying experimental logic becomes harder to discern. In this paper, we introduc…

Les travaux antérieurs sur lesquels il s’appuie

1. Très proche Pas pertinent

   Meta-Harness: End-to-End Optimization of Model Harnesses

   Yoonho Lee, R. Nair, Qizheng Zhang et al. · 2026 · arXiv.org · preprint, non relu par les pairs · cité 353 fois · Pas encore résumé

   Résumé scientifique

   The performance of large language model (LLM) systems depends not only on model weights, but also on their harness: the code that determines what information to store, retrieve, and present to the model. Yet harnesses are still designed largely by hand, and existing text optimizers are poorly matched to this setting because they compress feedback too aggressively. We introduce Meta-Harness, an outer-loop system that searches over harness code for LLM applications. It uses an agentic proposer that accesses the source code, scores, and execution traces of all prior candidates through a filesystem. On online text classification, Meta-Harness improves over a state-of-the-art context management s…

2. Apparenté Utilisé en profondeur par cet article Pas pertinent

   Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models

   Qizheng Zhang, Chang-Ran Hu, Shubhangi Upasani et al. · 2025 · arXiv.org · preprint, non relu par les pairs · cité 500 fois · Pas encore résumé

   Résumé scientifique

   Large language model (LLM) applications such as agents and domain-specific reasoning increasingly rely on context adaptation: modifying inputs with instructions, strategies, or evidence, rather than weight updates. Prior approaches improve usability but often suffer from brevity bias, which drops domain insights for concise summaries, and from context collapse, where iterative rewriting erodes details over time. We introduce ACE (Agentic Context Engineering), a framework that treats contexts as evolving playbooks that accumulate, refine, and organize strategies through a modular process of generation, reflection, and curation. ACE prevents collapse with structured, incremental updates that p…

3. Apparenté Pas pertinent

   Automated Design of Agentic Systems

   Sheng-Ran Hu, Cong Lu, Jeff Clune · 2024 · International Conference on Learning Representations · cité 494 fois · Pas encore résumé

   Résumé scientifique

   Researchers are investing substantial effort in developing powerful general-purpose agents, wherein Foundation Models are used as modules within agentic systems (e.g. Chain-of-Thought, Self-Reflection, Toolformer). However, the history of machine learning teaches us that hand-designed solutions are eventually replaced by learned solutions. We describe a newly forming research area, Automated Design of Agentic Systems (ADAS), which aims to automatically create powerful agentic system designs, including inventing novel building blocks and/or combining them in new ways. We further demonstrate that there is an unexplored yet promising approach within ADAS where agents can be defined in code and …

4. Apparenté Pas pertinent

   SWE-Hub: A Unified Production System for Scalable, Executable Software Engineering Tasks

   Yu-Chen Zeng, Shu-Peng Li, Da-Xiang Dong et al. · 2026 · arXiv.org · preprint, non relu par les pairs · cité 3 fois · Pas encore résumé

   Résumé scientifique

   Progress in software-engineering agents is increasingly constrained by the scarcity of executable, scalable, and realistic data for training and evaluation. This scarcity stems from three fundamental challenges in existing pipelines: environments are brittle and difficult to reproduce across languages; synthesizing realistic, system-level bugs at scale is computationally expensive; and existing data predominantly consists of short-horizon repairs, failing to capture long-horizon competencies like architectural consistency. We introduce \textbf{SWE-Hub}, an end-to-end system that operationalizes the data factory abstraction by unifying environment automation, scalable synthesis, and diverse t…

5. Apparenté Pas pertinent

   SkillClaw: Let Skills Evolve Collectively with Agentic Evolver

   Zi-Yu Ma, Shidong Yang, Yuxiang Ji et al. · 2026 · arXiv.org · preprint, non relu par les pairs · cité 77 fois · Pas encore résumé

   Résumé scientifique

   Large language model (LLM) agents such as OpenClaw rely on reusable skills to perform complex tasks, yet these skills remain largely static after deployment. As a result, similar workflows, tool usage patterns, and failure modes are repeatedly rediscovered across users, preventing the system from improving with experience. While interactions from different users provide complementary signals about when a skill works or fails, existing systems lack a mechanism to convert such heterogeneous experiences into reliable skill updates. To address these issues, we present SkillClaw, a framework for collective skill evolution in multi-user agent ecosystems, which treats cross-user and over-time inter…

Les articles qui le citent

1. Très proche Utilise cet article en profondeur Pas pertinent

   Rethinking the Evaluation of Harness Evolution for Agents

   Yike Wang, Huai-Sheng Zhu, Zheng-Yu Hu et al. · 2026 · arXiv.org · preprint, non relu par les pairs · cité 62 fois · Pas encore résumé

   Résumé scientifique

   Harness evolution is an iterative search procedure that repeatedly evaluates and revises candidate harnesses used for LLM agents using task feedback. We revisit the evaluation of such automatic harness evolution procedures and identify two fundamental issues in the protocol. First, prior work does not compare these approaches with simple task-level search baselines under matched feedback and inference budgets. Second, prior work searches for harness configurations using verification signals (e.g., unit test cases) drawn from the same benchmarks on which it reports the final performance of the evolved harnesses, violating the standard separation between training and test data. To address this…

2. Très proche Utilise cet article en profondeur Pas pertinent

   Harness-Delta Attribution: Dissecting Gains From Harness Evolution

   Wen-Xuan Ding, Qin Lu, Chang-Long Yu et al. · 2026 · Pas encore résumé

   Résumé scientifique

   Harness evolution automatically optimizes the prompts, skills, tools, and orchestration code around a fixed-weight model, offering a way to improve agentic systems without updating model weights. The agent harness is iteratively optimized to achieve higher performance on the search set. But how much of this gain reflects benchmark-specific overfitting or additional inference-time compute, and how much may generalize beyond the search set? In this work, we introduce Harness-Delta Attribution (HDA), an evaluation protocol that dissects an evolution gain into artifact-driven overfitting, gains explained by test-time scaling, and the remainder, which gives an upper bound on residual generalizabl…

3. Très proche Utilise cet article en profondeur Pas pertinent

   Hierarchical Self-Improvement: A Framework for Task-Specific Evolvable Agent Harnesses

   Tailin Zhou · 2026 · cité 9 fois · Pas encore résumé

   Résumé scientifique

   Modern LLM agents are often improved by modifying prompts, tools, or workflows manually, while the executable scaffold surrounding the model---the \emph{harness}---is typically treated as a fixed artifact after deployment. This work studies an alternative where the harness is \emph{task-specific and continuously evolvable}: each task family maintains its own harness, which is hot-swapped across iterations through a fixed task-injection seam and rewritten using environment feedback. We introduce \textbf{Hierarchical Self-Improvement (HSI)}, a framework in which a single frozen LLM $M$ operates across three hierarchical scopes: a task harness $H$ that executes tasks, an evolver that rewrites $…

4. Très proche Pas pertinent

   Harness-R1: Learning to Edit Executable Runtime Harnesses from Agent Failure Trajectories

   Shuai Shao, Kang-Ning Zhang, Qingyao Li et al. · 2026 · cité 19 fois · Pas encore résumé

   Résumé scientifique

   Agents built around large language models continually accumulate interaction trajectories during deployment, yet their behavior typically remains fixed. Beyond updating model weights, these trajectories can improve the agent harness that constructs context, mediates tools, validates actions, and recovers execution. We introduce Harness-R1, the first method, to our knowledge, that makes failure-conditioned, lifecycle-wide editing of an existing executable runtime a learned capability. It post-trains a dedicated harness engineer with online reinforcement learning so that its edits are optimized for the realized task success they produce, rather than proposed by a fixed editor. A separate 9B en…

5. Très proche Utilise cet article en profondeur Pas pertinent

   Verify Smarter, Evolve Further: Efficient Harness Evolution through Behavior-Aware Verification

   Jing-Heng Xu, Yi-Kai Zhang, Aiden Chen et al. · 2026 · cité 7 fois · Pas encore résumé

   Résumé scientifique

   Agent harnesses shape how language-model agents use instructions, tools, and runtime components, but adapting these harnesses requires costly verification. Existing propose-and-verify methods typically score every candidate on a fixed task set, wasting rollouts on unrelated behaviors and allowing aggregate scores to obscure specific regressions. We introduce HarnessLens, a budget-aware framework for automated harness evolution. HarnessLens jointly explores the task space and user-configurable components, derives candidate modifications from execution trajectories, and selectively verifies each candidate on behavior-relevant tasks using an attributable-evidence gate. Across three agent harnes…