Auditing Agent Harness Safety

Classés selon la proximité de leur titre et de leur résumé avec ceux de cet article. Sélectionnez-en pour les lire ensemble.

Les articles les plus proches

1. Très proche Cite cet article Pas pertinent

   HarnessRisk: A Lifecycle-Oriented Benchmark for Agent Harness Safety

   Ya Bai, Jinhao Duan, Jie Peng et al. · 2026 · cité 5 fois · Pas encore résumé

   Résumé scientifique

   Large language models are increasingly deployed through agent harnesses that manage tools, extensions, persistent state, permissions, and external actions. Existing safety benchmarks mainly target individual attack mechanisms or a limited subset of operational settings, making it difficult to compare how safety failures emerge across different harness responsibilities. We present HarnessRisk, a lifecycle oriented benchmark that organizes agent harness safety into six operational phases including Harness Configuration, Capability Extension, Runtime Operation, State Persistence, Action Control, and Incident Recovery. HarnessRisk contains 128 sandboxed cases, each pairing a benign user objectiv…

2. Très proche Pas pertinent

   Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows

   Yilun Yao, Xinyu Tan, Chao-Hsuan Liu et al. · 2026 · Artificial Intelligence · cité 3 fois · Résumé IA disponible

   Résumé scientifique

   LLM agents are increasingly deployed as executable systems that use tools, modify workspaces, and produce concrete artifacts. In such workflows, performance depends not only on the base model, but also on the harness: the system layer that manages context, tools, state, constraints, permissions, tracing, and recovery. However, existing benchmarks typically abstract away execution, compare complete agent systems, or hold the harness fixed, making execution-layer variation difficult to study. We introduce Harness-Bench, a diagnostic benchmark for evaluating configuration-level harness effects in realistic agent workflows. Harness-Bench evaluates representative harness configurations across mul…

3. Très proche Pas pertinent

   HarnessSafe: Evaluating Safety Across Persistent Carriers in Agent Harnesses

   Xiao Zhang, Yusheng Wang, Yuhao Fei et al. · 2026 · cité 5 fois · Pas encore résumé

   Résumé scientifique

   Modern agent harnesses persist state across tasks and sessions through persistent carriers like memory, skills, tools, and shared artifacts. However, this capability creates delayed safety risks: attacker-influenced content can cross system boundaries and later affect the execution of a benign request. Existing benchmarks typically focus on a few carriers or harnesses, while end-to-end attack-success rates reveal little about how risks propagate. To this end, we present HarnessSafe, a benchmark comprising 328 executable cases across seven persistent-carrier families and evaluated on most mainstream agent harnesses. Each case is specified as a Persistent-Risk Lifecycle that traces attacker in…

4. Très proche Pas pertinent

   From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flaws

   Meng-Zhuo Chen, Junjie Wang, Zhe Liu et al. · 2026 · arXiv.org · preprint, non relu par les pairs · cité 36 fois · Pas encore résumé

   Résumé scientifique

   LLM agents increasingly rely on agent harness: the runtime infrastructure around the base model that defines execution environments, tool interfaces, context, lifecycle orchestration, observability, verification, and governance. Existing self-improving agents and automatic harness evolution methods mainly improve agents through runtime supervision, prompt optimization, workflow search, or harness modification based on final outcomes. However, they often fail to diagnose where the responsible evidence lies in failed trajectories and which harness implementation mechanism causes the unreliable behavior, resulting in broad, indirect, or poorly scoped changes. This paper proposes HarnessFix, a t…

5. Très proche Cite cet article Pas pertinent

   HARDE: Optimizing Agent Harnesses for Runtime Risk Detection and Execution Control

   Zhuo Liu, Mo-Xin Li, Zhi-Xin Ma et al. · 2026 · Pas encore résumé

   Résumé scientifique

   Large language model (LLM) agents are vulnerable to safety risks such as injected malicious instructions or misleading information, motivating runtime defenses that prevent unsafe action in execution across diverse risks while preserving benign-task utility. Existing system-level defenses either focus on risk detection rather than timely prevention or rely on predefined rules with limited flexibility across diverse risks. We propose a risk-aware harness that integrates LLM-based monitoring for flexible risk detection and structures monitor-guided execution around three core modules: trigger, monitor, and feedback, enabling targeted safety interventions while limiting disruption to benign tas…

6. Très proche Cite cet article Pas pertinent

   Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution

   Zhi-Yong Cao, Heng-Jie Zhu, Yong-Xin Wang et al. · 2026 · cité 1 fois · Pas encore résumé

   Résumé scientifique

   LLM agents are increasingly used to solve real-world problems, and their effectiveness depends on both the underlying models and the harnesses that support their operation. However, most existing benchmarks focus on model capabilities and overall agent performance, with limited attention to the evaluation of agent harnesses. To address this gap, we introduce TraceTrial-Bench, which organizes harness reliability into seven components covering context, execution, governance, lifecycle, observability, tools, and verification. We construct 95 expert-level, mechanism-oriented tasks with executable or state-based ground truth to support semantically equivalent comparisons while preserving the nati…

7. Très proche Pas pertinent

   Stop Comparing LLM Agents Without Disclosing the Harness

   Yunbei Zhang, Janet Wang, Yingqiang Ge et al. · 2026 · Artificial Intelligence · preprint, non relu par les pairs · cité 44 fois · Résumé IA disponible

   Résumé scientifique

   This position paper argues that, for long-horizon tasks evaluated across models with comparable frontier capability, the agent execution harness, namely the infrastructure layer that governs context construction, tool interaction, orchestration, and verification around a language model, is often a stronger determinant of agent performance than the model it wraps. We formalize and defend the Binding Constraint Thesis: in this regime, performance variance is governed more by harness configuration than by model choice, and current evaluation protocols therefore systematically misattribute harness-level gains to model improvements. We support this thesis along three lines. First, a control-theor…

8. Très proche Cite cet article Pas pertinent

   Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents

   Yuyao Ge, Sheng-Hua Liu, Pei-Zhong Gao et al. · 2026 · Pas encore résumé

   Résumé scientifique

   Agents driven by large language models are increasingly deployed on long-horizon tasks, where they use tools and modify real workspaces. How well they perform depends as much on the harness around the model as on the model itself. Existing benchmarks measure downstream task success under a fixed harness, leaving harness engineering itself unexplored. We propose Doc2Harness, a two-stage benchmark that evaluates long-horizon harness engineering directly. In Stage 1, a build model operating inside a fixed builder harness receives an interface-only contract skeleton and a development document that consolidates behavior across twelve frontier harnesses, and then constructs a complete candidate ha…

Les articles qui le citent

1. Proche Pas pertinent

   AgentS4D: Benchmarking Runtime Risks across the Execution Lifecycle of LLM-Based Workspace Agents

   Jiajun Zhou, Zhaoxuan Ke, Ji-Hang Ye et al. · 2026 · arXiv.org · preprint, non relu par les pairs · cité 2 fois · Pas encore résumé

   Résumé scientifique

   Large language model (LLM)-based workspace agents execute stateful, multi-step workflows across heterogeneous resources, external tools, and persistent state. Their safety must therefore be assessed from actions, side effects, and state changes throughout execution. Although recent benchmarks have advanced executable safety testing and trajectory-aware verification, they rarely provide a unified account of where risks enter, how they elicit unsafe behavior, which harms they target, and where supporting evidence appears during execution. We introduce AgentS4D, a sandboxed benchmark for lifecycle-wide runtime safety evaluation. Its four-dimensional runtime-safety framework uses six risk-entry …

2. Proche Pas pertinent

   SoK: When Safe Agents Fail Together: The Security of Multi Agent LLM Systems

   Rui Yang, Jun-Jie Xu, Zheng-Yu Liu et al. · 2026 · cité 4 fois · Pas encore résumé

   Résumé scientifique

   Safe agents can fail together. Multi-agent LLM systems (MAS) move information, state, decisions, and authority across principal boundaries, creating failures that local checks may miss. Without an execution-level view, a multi-agent setting can easily be mistaken for evidence of a genuinely multi-agent security effect. We thus systematize MAS security through an execution-centered analysis of 197 works, covering six interaction interfaces, four adversary positions, seven system-level risks, and eight recurring attack paths. We introduce an A-I-R framework that organizes attacks by adversary position, interaction interface, and resulting system-level risk, unifying otherwise fragmented attack…

3. Proche Pas pertinent

   Trust the Brand, Lose Control: How Identity Hijacks LLM Agent Orchestration

   Xu-Tao Mao, Rui Qian, Ling-Han Chen et al. · 2026 · cité 1 fois · Pas encore résumé

   Résumé scientifique

   LLM agents now execute tasks end to end with permission to change real systems and increasingly orchestrate subagents that differ in capability and cost. Prior work treats the choice of subagent as an optimization problem. Yet the orchestrator makes this choice from the identities that subagents display, and an attacker can spoof them. Displayed identity thus decides operational authority, meaning who is trusted to check the work and who is allowed to change it. As a result, a risky subagent can keep authority over execution even after other evidence contradicts it. We introduce TrustFork, an LLM agent safety benchmark with 1,890 tasks and 27,826 valid trajectories across 16 agent systems. T…

4. Apparenté Pas pertinent

   Reachability-Based Capability Confinement for LLM Agents under Indirect Prompt Injection

   Wu-Jie Xiong, Rabimba Karanjai, Yang Lu et al. · 2026 · cité 3 fois · Pas encore résumé

   Résumé scientifique

   Large language model agents place outputs from external skills into their execution context, allowing attacker-controlled data to influence later privileged actions. Existing defenses mainly classify untrusted content or authorize proposed operations. They do not directly address how an agent's future authority should change once untrusted data enters its state. We present SkillGuard, a harness-level enforcement layer that treats this event as contamination and restricts future capabilities to disconnect the resulting state from deployer-defined forbidden states. Given sound skill summaries and policies, SkillGuard represents security-relevant transitions with a Skill Impact Graph, specifies…

5. Apparenté Utilise cet article en profondeur Pas pertinent

   Soei: Structured Observability of Evidence and Intent for Multi-Agent AI Software Engineering

   Mohammed Nayeem Khan, K. S. Sarun Kumar · 2026 · Pas encore résumé

   Résumé scientifique

   AI coding agents now do long, multi-step and multi-session software work with little supervision. When a developer comes back to that work, the useful question is not what the model said but what was actually done, by whom, and whether it is finished. We study this in three parts. First, we verify Agenttrail, an open-source local observability tool for coding agents, against its source at commit e4ba2da: nine documented constants and mechanisms hold at file-and-line level, two event pipelines are never unified, and its Map service has an unauthenticated local mutation surface. Second, we run a prototype of our system on 45 real coding-agent sessions (102,233 records, 20,577 tool executions, …