Do AI-for-Science Agents Preserve Model Safety?

Classés selon la proximité de leur titre et de leur résumé avec ceux de cet article. Sélectionnez-en pour les lire ensemble.

Les articles les plus proches

1. Très proche Pas pertinent

   SafeScientist: Enhancing AI Scientist Safety for Risk-Aware Scientific Discovery

   Kun-Lun Zhu, Jiaxun Zhang, Ziheng Qi et al. · 2025 · Conference on Empirical Methods in Natural Language Processing · Pas encore résumé

   Résumé scientifique

   Recent advancements in large language model (LLM) agents have significantly accelerated scientific discovery automation, yet concurrently raised critical ethical and safety concerns. To systematically address these challenges, we introduce SafeScientist , an innovative AI scientist framework explicitly designed to enhance safety and ethical responsibility in AI-driven scientific exploration. SafeScientist proactively refuses ethically inappropriate or high-risk tasks and rigorously emphasizes safety throughout the research process. To achieve comprehensive safety oversight, we integrate multiple defensive mechanisms, including prompt monitoring, agent-collaboration monitoring, tool-use monit…

2. Très proche Pas pertinent

   Self-Harness: Harnesses That Improve Themselves

   Hangfan Zhang, Shao Zhang, Kangcong Li et al. · 2026 · Computation and Language · Résumé IA disponible

   Résumé scientifique

   The performance of LLM-based agents is jointly shaped by their base models and the harnesses that mediate their interaction with the environment. Because different models exhibit distinct behaviors, effective harness design is inherently model-specific. Yet agent harnesses are still largely engineered by human experts, a paradigm that scales poorly as modern LLMs become increasingly diverse and rapidly evolving. In this paper, we introduce Self-Harness, a new paradigm in which an LLM-based agent improves its own operating harness, without relying on human engineers or stronger external agents. We operationalize Self-Harness as an iterative loop with three stages: Weakness Mining, which ident…

3. Très proche Cité par cet article Pas pertinent

   SafeScientist: Toward Risk-Aware Scientific Discoveries by LLM Agents

   Kun-Lun Zhu, Jiaxun Zhang, Ziheng Qi et al. · 2025 · arXiv.org · Pas encore résumé

   Résumé scientifique

   Recent advancements in large language model (LLM) agents have significantly accelerated scientific discovery automation, yet concurrently raised critical ethical and safety concerns. To systematically address these challenges, we introduce \textbf{SafeScientist}, an innovative AI scientist framework explicitly designed to enhance safety and ethical responsibility in AI-driven scientific exploration. SafeScientist proactively refuses ethically inappropriate or high-risk tasks and rigorously emphasizes safety throughout the research process. To achieve comprehensive safety oversight, we integrate multiple defensive mechanisms, including prompt monitoring, agent-collaboration monitoring, tool-u…

4. Proche Pas pertinent

   Tool Specifications Matter: Uncovering and Mitigating Safety Risks in AI Agents

   Minghui Pan, Jiayuxuan Yang, Yuan-Yuan Yuan et al. · 2026 · arXiv.org · Pas encore résumé

   Résumé scientifique

   AI agents extend large language models (LLMs) with external tools, enabling them to perform complex tasks and translate model outputs into consequential real-world actions. Yet LLMs often become substantially less safe when deployed as agents, and the source of this degradation remains poorly understood. In this paper, we identify schema-formatted tool specifications as a primary source of agent safety degradation and show, through white-box representation analysis, that they weaken the model's internal refusal signals and contribute to unsafe tool execution. Building on this finding, we propose SafeKeep, an inference-time safeguard that decouples safety judgment from tool execution: it asse…

5. Proche Pas pertinent

   Natural-Language Agent Harnesses

   Linyue Pan, Lexiao Zou, Shuo Guo et al. · 2026 · Computation and Language · Résumé IA disponible

   Résumé scientifique

   Agent performance is strongly shaped by the surrounding harness: the external execution system around a model that organizes a task run. Yet this logic is usually buried in tightly coupled controller code, which makes harnesses hard to inspect, compare, transfer, and ablate. This paper asks whether the reusable design pattern of an agent harness can be represented as an executable natural-language object. We introduce Natural-Language Agent Harnesses (NLAHs), editable documents that describe run-level harness policy, and Intelligent Harness Runtime (IHR), a shared runtime that interprets these documents into agent calls, handoffs, state updates, validation gates, and artifact contracts. Acro…

6. Proche Pas pertinent

   Trustworthy LLM-Based Agents for Data-Centric AI

   Tian-Long Chen, Jian Pei, Minxing Zhang et al. · 2026 · Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 · Pas encore résumé

   Résumé scientifique

   Large language model (LLM) based agents are evolving from conversational chatbots into autonomous decision-makers that plan, reason, wield tools, and collaborate across high-stakes domains such as healthcare, finance, and scientific discovery. Yet this power brings a fundamental challenge: trustworthiness. How can we guarantee that an agent remains robust when real-world data shifts, degrades, or is deliberately poisoned? What defenses exist against memory injection, tool-based exploits, or cascade failures in multi-agent systems? Can we embed domain-specific causal validity, clinical safety, or fairness directly into agent reasoning? And how do we measure trust when it spans robustness, sec…

7. Proche Pas pertinent

   Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training

   Evan Hubinger, Carson Denison, Jesse Mu et al. · 2024 · Cryptography and Security · Résumé IA disponible

   Résumé scientifique

   Humans are capable of strategically deceptive behavior: behaving helpfully in most situations, but then behaving very differently in order to pursue alternative objectives when given the opportunity. If an AI system learned such a deceptive strategy, could we detect it and remove it using current state-of-the-art safety training techniques? To study this question, we construct proof-of-concept examples of deceptive behavior in large language models (LLMs). For example, we train models that write secure code when the prompt states that the year is 2023, but insert exploitable code when the stated year is 2024. We find that such backdoored behavior can be made persistent, so that it is not rem…

8. Proche Pas pertinent

   Toward Reliable, Safe, and Secure LLMs for Scientific Applications

   S. S. Chaturvedi, J. Bergerson, Tanwi Mallick · 2026 · arXiv.org · Pas encore résumé

   Résumé scientifique

   As large language models (LLMs) evolve into autonomous"AI scientists,"they promise transformative advances but introduce novel vulnerabilities, from potential"biosafety risks"to"dangerous explosions."Ensuring trustworthy deployment in science requires a new paradigm centered on reliability (ensuring factual accuracy and reproducibility), safety (preventing unintentional physical or biological harm), and security (preventing malicious misuse). Existing general-purpose safety benchmarks are poorly suited for this purpose, suffering from a fundamental domain mismatch, limited threat coverage of science-specific vectors, and benchmark overfitting, which create a critical gap in vulnerability eva…

Les travaux antérieurs sur lesquels il s’appuie

1. Proche Pas pertinent

   AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents

   Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian et al. · 2024 · International Conference on Learning Representations · Pas encore résumé

   Résumé scientifique

   The robustness of LLMs to jailbreak attacks, where users design prompts to circumvent safety measures and misuse model capabilities, has been studied primarily for LLMs acting as simple chatbots. Meanwhile, LLM agents -- which use external tools and can execute multi-stage tasks -- may pose a greater risk if misused, but their robustness remains underexplored. To facilitate research on LLM agent misuse, we propose a new benchmark called AgentHarm. The benchmark includes a diverse set of 110 explicitly malicious agent tasks (440 with augmentations), covering 11 harm categories including fraud, cybercrime, and harassment. In addition to measuring whether models refuse harmful agentic requests,…

2. Proche Pas pertinent

   SafeSci: Safety Evaluation of Large Language Models in Science Domains and Beyond

   Xiang-Yang Zhu, Yuan Tian, Qi Jia et al. · 2026 · arXiv.org · Pas encore résumé

   Résumé scientifique

   The success of large language models (LLMs) in scientific domains has heightened safety concerns, prompting numerous benchmarks to evaluate their scientific safety. Existing benchmarks often suffer from limited risk coverage and a reliance on subjective evaluation. To address these problems, we introduce SafeSci, a comprehensive framework for safety evaluation and enhancement in scientific contexts. SafeSci comprises SafeSciBench, a multi-disciplinary benchmark with 0.25M samples, and SafeSciTrain, a large-scale dataset containing 1.5M samples for safety enhancement. SafeSciBench distinguishes between safety knowledge and risk to cover extensive scopes and employs objective metrics such as d…

3. Proche Pas pertinent

   SoSBench: Benchmarking Safety Alignment on Six Scientific Domains

   Feng-Qing Jiang, Feng-Bo Ma, Zhangchen Xu et al. · 2025 · Pas encore résumé

   Résumé scientifique

   Large language models (LLMs) exhibit advancing capabilities in complex tasks, such as reasoning and graduate-level question answering, yet their resilience against misuse, particularly involving scientifically sophisticated risks, remains underexplored. Existing safety benchmarks typically focus either on instructions requiring minimal knowledge comprehension (e.g., ``tell me how to build a bomb") or utilize prompts that are relatively low-risk (e.g., multiple-choice or classification tasks about hazardous content). Consequently, they fail to adequately assess model safety when handling knowledge-intensive, hazardous scenarios. To address this critical gap, we introduce SoSBench, a regulatio…

4. Proche Pas pertinent

   Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents

   Hanrong Zhang, Jingyuan Huang, Kai Mei et al. · 2024 · International Conference on Learning Representations · Pas encore résumé

   Résumé scientifique

   Although LLM-based agents, powered by Large Language Models (LLMs), can use external tools and memory mechanisms to solve complex real-world tasks, they may also introduce critical security vulnerabilities. However, the existing literature does not comprehensively evaluate attacks and defenses against LLM-based agents. To address this, we introduce Agent Security Bench (ASB), a comprehensive framework designed to formalize, benchmark, and evaluate the attacks and defenses of LLM-based agents, including 10 scenarios (e.g., e-commerce, autonomous driving, finance), 10 agents targeting the scenarios, over 400 tools, 27 different types of attack/defense methods, and 7 evaluation metrics. Based o…

5. Apparenté Pas pertinent

   Identifying the Risks of LM Agents with an LM-Emulated Sandbox

   Yangjun Ruan, Honghua Dong, Andrew Wang et al. · 2023 · International Conference on Learning Representations · Pas encore résumé

   Résumé scientifique

   Recent advances in Language Model (LM) agents and tool use, exemplified by applications like ChatGPT Plugins, enable a rich set of capabilities but also amplify potential risks - such as leaking private data or causing financial losses. Identifying these risks is labor-intensive, necessitating implementing the tools, setting up the environment for each test scenario manually, and finding risky cases. As tools and agents become more complex, the high cost of testing these agents will make it increasingly difficult to find high-stakes, long-tailed risks. To address these challenges, we introduce ToolEmu: a framework that uses an LM to emulate tool execution and enables the testing of LM agents…