System Prompts Affect Response Bias More Than Discriminability in LLM Safety
Auteurs : Mohan W. Gupta, Sandy Tanwisuth
Résumé scientifique
Large language model safety evaluations often focus on a compliance-refusal tradeoff: models should refuse unsafe requests without refusing safe ones. We argue that this misses a broader form of prompt-conditioned task selection, where the same request can elicit different behaviors depending on the system prompt. We investigate this using a controlled evaluation crossing eight safety-task prompt families with three clarity levels across HarmBench, XSTest, and IHEval. Holding benchmark items fixed, we evaluate 13 models and use signal detection theory to distinguish two possibilities: whether models become better at telling safe from unsafe requests (discriminability), or simply more willing to refuse (response criterion). We find substantial differences in discriminability across models, but system prompts primarily shift the willingness to refuse, with smaller and less consistent effects on discriminability. Increasing prompt clarity makes models more willing to refuse without reliably improving their ability to distinguish safe from unsafe requests. Beyond refusal, prompt families also shift responses among six distinct behavioral policies, including clarification and safe help. Together, these findings show that system prompts primarily change how models respond to safety-relevant requests rather than consistently improving their ability to distinguish safe from unsafe ones.
Les articles autour de lui, par date et par proximité
Plus haut, c’est plus proche de cet article ; à gauche, c’est plus ancien. Les traits relient les articles proches entre eux : les groupes de traits sont des sous-thèmes. Sélectionnez un point pour voir pourquoi il est là. Retour au résumé de cet article
Les articles autour de celui-ci
Classés selon la proximité de leur titre et de leur résumé avec ceux de cet article. Sélectionnez-en pour les lire ensemble.
Les articles les plus proches
-
The Refusal-Compliance Tradeoff: A Large-Scale Safety Behavior Audit of Large Language ModelsAlif Al Hasan, Sumon Biswas · 2026 · arXiv.org · preprint, non relu par les pairs · cité 1 fois · Pas encore résumé
The Refusal-Compliance Tradeoff audits how models balance refusals of harmful requests against over-refusals of benign ones across safety benchmarks.
Résumé scientifique
Refusal rates are a poor proxy for LLM safety, i.e., a model may over-refuse benign prompts while still complying with harmful ones. We audit both failure modes across 21 open-weight LLMs on four safety benchmarks (OR-Bench, XSTest, ToxiGen, BOLD), using a composition adjustment to isolate model sensitivity from dataset toxicity confounds. We report three findings. First, models adopt fundamentally different calibration strategies: conservative ecosystems such as Llama suppress unsafe outputs at the cost of elevated over-refusals, while permissive ecosystems such as DeepSeek and Qwen preserve helpfulness but tolerate higher harmful compliance. Second, demographic protection is unequal: model…
-
Mitigating Exaggerated Safety in Large Language ModelsRuchira Ray, Ruchi Bhalani · 2024 · arXiv.org · preprint, non relu par les pairs · cité 10 fois · Pas encore résumé
Mitigating Exaggerated Safety tests interactive, contextual, and few-shot prompting to reduce refusals of safe requests, rather than measuring prompt-conditioned discriminability.
Résumé scientifique
As the popularity of Large Language Models (LLMs) grow, combining model safety with utility becomes increasingly important. The challenge is making sure that LLMs can recognize and decline dangerous prompts without sacrificing their ability to be helpful. The problem of"exaggerated safety"demonstrates how difficult this can be. To reduce excessive safety behaviours -- which was discovered to be 26.1% of safe prompts being misclassified as dangerous and refused -- we use a combination of XSTest dataset prompts as well as interactive, contextual, and few-shot prompting to examine the decision bounds of LLMs such as Llama2, Gemma Command R+, and Phi-3. We find that few-shot prompting works best…
-
From Prompt Risk to Response Risk: Paired Analysis of Safety Behavior of Large Language ModelMeng-Ya Hu, Qiong Wei, Sandeep Atluri · 2026 · arXiv.org · preprint, non relu par les pairs · Pas encore résumé
From Prompt Risk to Response Risk measures how model responses change the severity of harm, whereas this paper studies how system prompts shape safety behaviors.
Résumé scientifique
Safety evaluations of large language models (LLMs) typically report binary outcomes, i.e. attack success rate (ASR), refusal rate, or harmful versus safe classification, which hide how risk changes between prompt and response. We present a paired analysis over human labeled prompt and response records across four harm categories (Sexual, Self harm, Hate and Violence) and ordinal severity levels (Safe, Low, Medium, High). 61% of responses reduce harm relative to the prompt, 36% preserve severity, and 3% escalate. The escalation splits into two mechanisms: benign prompts triggering unrequested harmful detail, and answers that stay on task at higher severity than the prompt. Category decomposit…
-
Resisting Correction: How RLHF Makes Language Models Ignore External Safety Signals in Natural ConversationFelipe Biava Cataneo · 2025 · arXiv.org · preprint, non relu par les pairs · cité 2 fois · Pas encore résumé
Resisting Correction tests whether external confidence signals change model answers on GSM8K, not how safety prompts affect refusal decisions.
Résumé scientifique
Safety architectures for language models increasingly rely on external monitors to detect errors and inject corrective signals at inference time. For such systems to function in interactive settings, models must be able to incorporate externally provided confidence information into their verbal responses. In this work, we test whether instruction-tuned language models preserve this controllability across different interaction modes. Using Llama-3.2-3B on GSM8K, we perform a causal intervention study in which explicit external confidence signals are injected and model compliance is measured under multiple prompt strategies. We find that base models exhibit near-perfect controllability (Spearm…
-
The Instability of Safety: How Random Seeds and Temperature Expose Inconsistent LLM Refusal BehaviorE. Larsen · 2025 · arXiv.org · preprint, non relu par les pairs · cité 7 fois · Pas encore résumé
The Instability of Safety examines changes in refusal decisions across sampling settings, while this paper tests how system prompts shift refusal behavior.
Résumé scientifique
Current safety evaluations of large language models rely on single-shot testing, implicitly assuming that model responses are deterministic and representative of the model's safety alignment. We challenge this assumption by investigating the stability of safety refusal decisions across random seeds and temperature settings. Testing four instruction-tuned models from three families (Llama 3.1 8B, Qwen 2.5 7B, Qwen 3 8B, Gemma 3 12B) on 876 harmful prompts across 20 sampling configurations (4 temperatures x 5 seeds), we find that 18-28% of prompts exhibit decision flips (the REFUSE/PARTIAL/COMPLY label changes across configurations), depending on the model. Our Safety Stability Index (SSI) rev…
-
Are Large Language Models Good Prompt Optimizers?Ruotian Ma, Xiaolei Wang, Xin Zhou et al. · 2024 · Computation and Language · preprint, non relu par les pairs · cité 46 fois · Résumé IA disponible
Are Large Language Models Good Prompt Optimizers studies how models refine prompts and diagnose errors, rather than how safety prompts alter responses.
Résumé scientifique
LLM-based Automatic Prompt Optimization, which typically utilizes LLMs as Prompt Optimizers to self-reflect and refine prompts, has shown promising performance in recent studies. Despite the success, the underlying mechanism of this approach remains unexplored, and the true effectiveness of LLMs as Prompt Optimizers requires further validation. In this work, we conducted a comprehensive study to uncover the actual mechanism of LLM-based Prompt Optimization. Our findings reveal that the LLM optimizers struggle to identify the true causes of errors during reflection, tending to be biased by their own prior knowledge rather than genuinely reflecting on the errors. Furthermore, even when the ref…
-
Beyond Over-Refusal: Scenario-Based Diagnostics and Post-Hoc Mitigation for Exaggerated Refusals in LLMsShu-Zhou Yuan, Ercong Nie, Yin Sun et al. · 2025 · arXiv.org · preprint, non relu par les pairs · cité 5 fois · Pas encore résumé
Beyond Over-Refusal introduces single- and multi-turn benchmarks and mitigation for false refusals, instead of analyzing prompt effects on refusal criteria.
Résumé scientifique
Large language models (LLMs) frequently produce false refusals, declining benign requests that contain terms resembling unsafe queries. We address this challenge by introducing two comprehensive benchmarks: the Exaggerated Safety Benchmark (XSB) for single-turn prompts, annotated with"Focus"keywords that identify refusal-inducing triggers, and the Multi-turn Scenario-based Exaggerated Safety Benchmark (MS-XSB), which systematically evaluates refusal calibration in realistic, context-rich dialog settings. Our benchmarks reveal that exaggerated refusals persist across diverse recent LLMs and are especially pronounced in complex, multi-turn scenarios. To mitigate these failures, we leverage pos…
-
When Prompt Optimization Becomes Jailbreaking: Adaptive Red-Teaming of Large Language ModelsZafir Shamsi, Nikhil Chekuru, Z. Guzman et al. · 2026 · Conference of the European Chapter of the Association for Computational Linguistics · cité 1 fois · Pas encore résumé
When Prompt Optimization Becomes Jailbreaking uses adaptive prompt refinement to find safety failures, whereas this paper compares fixed safety instructions.
Résumé scientifique
Large Language Models (LLMs) are increasingly integrated into high-stakes applications, making robust safety guarantees a central practical and commercial concern. Existing safety evaluations predominantly rely on fixed collections of harmful prompts, implicitly assuming non-adaptive adversaries and thereby overlooking realistic attack scenarios in which inputs are iteratively refined to evade safeguards. In this work, we examine the vulnerability of contemporary language models to automated, adversarial prompt refinement. We repurpose black-box prompt optimization techniques, originally designed to improve performance on benign tasks, to systematically search for safety failures. Using DSPy…
Les travaux antérieurs sur lesquels il s’appuie
-
LLMs as Signal Detectors: Sensitivity, Bias, and the Temperature-Criterion AnalogyJon-Paul Cacioli · 2026 · arXiv.org · preprint, non relu par les pairs · cité 9 fois · Pas encore résumé
LLMs as Signal Detectors applies signal detection theory to separate sensitivity from response bias, the distinction this paper uses to analyze refusals.
Résumé scientifique
Large language models (LLMs) are evaluated for calibration using metrics such as Expected Calibration Error that conflate two distinct components: the model's ability to discriminate correct from incorrect answers (sensitivity) and its tendency toward confident or cautious responding (bias). Signal Detection Theory (SDT) decomposes these components. While SDT-derived metrics such as AUROC are increasingly used, the full parametric framework - unequal-variance model fitting, criterion estimation, z-ROC analysis - has not been applied to LLMs as signal detectors. In this pre-registered study, we treat three LLMs as observers performing factual discrimination across 168,000 trials and test whet…
-
G-Eval: NLG Evaluation using GPT-4 with Better Human AlignmentYang Liu, Dan Iter, Yi-Chong Xu et al. · 2023 · Conference on Empirical Methods in Natural Language Processing · cité 3 690 fois · Résumé IA disponible
G-Eval uses an LLM as a text evaluator, a generic evaluation method used by this paper to classify behavioral policies.
Résumé scientifique
The quality of texts generated by natural language generation (NLG) systems is hard to measure automatically. Conventional reference-based metrics, such as BLEU and ROUGE, have been shown to have relatively low correlation with human judgments, especially for tasks that require creativity and diversity. Recent studies suggest using large language models (LLMs) as reference-free metrics for NLG evaluation, which have the benefit of being applicable to new tasks that lack human references. However, these LLM-based evaluators still have lower human correspondence than medium-size neural evaluators. In this work, we present G-Eval, a framework of using large language models with chain-of-thought…
Aller plus loin
Comment ces articles sont-ils trouvés ?
Nous comparons le titre et le résumé de cet article avec ceux de tous les articles résumés sur ce site, et avec les articles que Semantic Scholar recommande, ceux qu’il cite et ceux qui le citent (pour un article très cité, ses citations les plus récentes et, depuis OpenAlex, les plus citées). Parmi les travaux antérieurs et les suites, les articles qui s’utilisent en profondeur, une méthode qu’il utilise et les suites très citées remontent un peu. Les niveaux (très proche, proche, apparenté) viennent de la similarité des textes, pas d’un jugement de qualité ; * signale un article sans résumé, comparé par son titre. La phrase sous un article est écrite par une IA à partir des deux résumés : vérifiez-la dans les articles. Un article jugé hors sujet par trois lecteurs quitte la liste.
Mis à jour le 16 novembre 2026.