System Prompts Affect Response Bias More Than Discriminability in LLM Safety
Authors: Mohan W. Gupta, Sandy Tanwisuth
Abstract
Large language model safety evaluations often focus on a compliance-refusal tradeoff: models should refuse unsafe requests without refusing safe ones. We argue that this misses a broader form of prompt-conditioned task selection, where the same request can elicit different behaviors depending on the system prompt. We investigate this using a controlled evaluation crossing eight safety-task prompt families with three clarity levels across HarmBench, XSTest, and IHEval. Holding benchmark items fixed, we evaluate 13 models and use signal detection theory to distinguish two possibilities: whether models become better at telling safe from unsafe requests (discriminability), or simply more willing to refuse (response criterion). We find substantial differences in discriminability across models, but system prompts primarily shift the willingness to refuse, with smaller and less consistent effects on discriminability. Increasing prompt clarity makes models more willing to refuse without reliably improving their ability to distinguish safe from unsafe requests. Beyond refusal, prompt families also shift responses among six distinct behavioral policies, including clarification and safe help. Together, these findings show that system prompts primarily change how models respond to safety-relevant requests rather than consistently improving their ability to distinguish safe from unsafe ones.
The papers around it, by date and closeness
Higher is closer to this paper; left is older. Lines join papers close to each other: groups of lines are sub-topics. Select a point to see why it is there. Back to the summary of this paper
Papers around this one
Ranked by how close their title and abstract are to this paper’s. Select some to read them together.
Closest papers
-
The Refusal-Compliance Tradeoff: A Large-Scale Safety Behavior Audit of Large Language ModelsAlif Al Hasan, Sumon Biswas · 2026 · arXiv.org · preprint, not peer reviewed · cited 1 time · Not summarized yet
The audit compares harmful compliance with benign over-refusal across safety benchmarks, examining the refusal-compliance balance that this paper separates into discriminability and refusal tendency.
Abstract
Refusal rates are a poor proxy for LLM safety, i.e., a model may over-refuse benign prompts while still complying with harmful ones. We audit both failure modes across 21 open-weight LLMs on four safety benchmarks (OR-Bench, XSTest, ToxiGen, BOLD), using a composition adjustment to isolate model sensitivity from dataset toxicity confounds. We report three findings. First, models adopt fundamentally different calibration strategies: conservative ecosystems such as Llama suppress unsafe outputs at the cost of elevated over-refusals, while permissive ecosystems such as DeepSeek and Qwen preserve helpfulness but tolerate higher harmful compliance. Second, demographic protection is unequal: model…
-
Mitigating Exaggerated Safety in Large Language ModelsRuchira Ray, Ruchi Bhalani · 2024 · arXiv.org · preprint, not peer reviewed · cited 10 times · Not summarized yet
Mitigating exaggerated refusals uses interactive, contextual, and few-shot prompting on XSTest to reduce refusals of safe requests, rather than measuring system-prompt effects with signal detection theory.
Abstract
As the popularity of Large Language Models (LLMs) grow, combining model safety with utility becomes increasingly important. The challenge is making sure that LLMs can recognize and decline dangerous prompts without sacrificing their ability to be helpful. The problem of"exaggerated safety"demonstrates how difficult this can be. To reduce excessive safety behaviours -- which was discovered to be 26.1% of safe prompts being misclassified as dangerous and refused -- we use a combination of XSTest dataset prompts as well as interactive, contextual, and few-shot prompting to examine the decision bounds of LLMs such as Llama2, Gemma Command R+, and Phi-3. We find that few-shot prompting works best…
-
From Prompt Risk to Response Risk: Paired Analysis of Safety Behavior of Large Language ModelMeng-Ya Hu, Qiong Wei, Sandeep Atluri · 2026 · arXiv.org · preprint, not peer reviewed · Not summarized yet
The paired analysis tracks how model responses change harm severity relative to prompts, addressing safety behavior beyond the refusal-versus-compliance outcomes studied here.
Abstract
Safety evaluations of large language models (LLMs) typically report binary outcomes, i.e. attack success rate (ASR), refusal rate, or harmful versus safe classification, which hide how risk changes between prompt and response. We present a paired analysis over human labeled prompt and response records across four harm categories (Sexual, Self harm, Hate and Violence) and ordinal severity levels (Safe, Low, Medium, High). 61% of responses reduce harm relative to the prompt, 36% preserve severity, and 3% escalate. The escalation splits into two mechanisms: benign prompts triggering unrequested harmful detail, and answers that stay on task at higher severity than the prompt. Category decomposit…
-
Resisting Correction: How RLHF Makes Language Models Ignore External Safety Signals in Natural ConversationFelipe Biava Cataneo · 2025 · arXiv.org · preprint, not peer reviewed · cited 2 times · Not summarized yet
The intervention study tests whether models follow external confidence signals under different prompt strategies, but studies correction in math responses rather than safety-request discrimination.
Abstract
Safety architectures for language models increasingly rely on external monitors to detect errors and inject corrective signals at inference time. For such systems to function in interactive settings, models must be able to incorporate externally provided confidence information into their verbal responses. In this work, we test whether instruction-tuned language models preserve this controllability across different interaction modes. Using Llama-3.2-3B on GSM8K, we perform a causal intervention study in which explicit external confidence signals are injected and model compliance is measured under multiple prompt strategies. We find that base models exhibit near-perfect controllability (Spearm…
-
The Instability of Safety: How Random Seeds and Temperature Expose Inconsistent LLM Refusal BehaviorE. Larsen · 2025 · arXiv.org · preprint, not peer reviewed · cited 7 times · Not summarized yet
The study measures how sampling seeds and temperature change refusal decisions on harmful prompts, whereas this paper tests how system prompts shift safety behavior.
Abstract
Current safety evaluations of large language models rely on single-shot testing, implicitly assuming that model responses are deterministic and representative of the model's safety alignment. We challenge this assumption by investigating the stability of safety refusal decisions across random seeds and temperature settings. Testing four instruction-tuned models from three families (Llama 3.1 8B, Qwen 2.5 7B, Qwen 3 8B, Gemma 3 12B) on 876 harmful prompts across 20 sampling configurations (4 temperatures x 5 seeds), we find that 18-28% of prompts exhibit decision flips (the REFUSE/PARTIAL/COMPLY label changes across configurations), depending on the model. Our Safety Stability Index (SSI) rev…
-
Are Large Language Models Good Prompt Optimizers?Ruotian Ma, Xiaolei Wang, Xin Zhou et al. · 2024 · Computation and Language · preprint, not peer reviewed · cited 46 times · AI summary available
The study investigates how LLMs optimize prompts for task performance, rather than how system prompts alter safety responses.
Abstract
LLM-based Automatic Prompt Optimization, which typically utilizes LLMs as Prompt Optimizers to self-reflect and refine prompts, has shown promising performance in recent studies. Despite the success, the underlying mechanism of this approach remains unexplored, and the true effectiveness of LLMs as Prompt Optimizers requires further validation. In this work, we conducted a comprehensive study to uncover the actual mechanism of LLM-based Prompt Optimization. Our findings reveal that the LLM optimizers struggle to identify the true causes of errors during reflection, tending to be biased by their own prior knowledge rather than genuinely reflecting on the errors. Furthermore, even when the ref…
-
Beyond Over-Refusal: Scenario-Based Diagnostics and Post-Hoc Mitigation for Exaggerated Refusals in LLMsShu-Zhou Yuan, Ercong Nie, Yin Sun et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 5 times · Not summarized yet
The work diagnoses and mitigates false refusals with single- and multi-turn benchmarks, focusing on exaggerated refusal rather than prompt effects on discriminability.
Abstract
Large language models (LLMs) frequently produce false refusals, declining benign requests that contain terms resembling unsafe queries. We address this challenge by introducing two comprehensive benchmarks: the Exaggerated Safety Benchmark (XSB) for single-turn prompts, annotated with"Focus"keywords that identify refusal-inducing triggers, and the Multi-turn Scenario-based Exaggerated Safety Benchmark (MS-XSB), which systematically evaluates refusal calibration in realistic, context-rich dialog settings. Our benchmarks reveal that exaggerated refusals persist across diverse recent LLMs and are especially pronounced in complex, multi-turn scenarios. To mitigate these failures, we leverage pos…
-
When Prompt Optimization Becomes Jailbreaking: Adaptive Red-Teaming of Large Language ModelsZafir Shamsi, Nikhil Chekuru, Z. Guzman et al. · 2026 · Conference of the European Chapter of the Association for Computational Linguistics · cited 1 time · Not summarized yet
The work adaptively refines adversarial prompts to find safety failures, rather than comparing system prompts on fixed safety requests.
Abstract
Large Language Models (LLMs) are increasingly integrated into high-stakes applications, making robust safety guarantees a central practical and commercial concern. Existing safety evaluations predominantly rely on fixed collections of harmful prompts, implicitly assuming non-adaptive adversaries and thereby overlooking realistic attack scenarios in which inputs are iteratively refined to evade safeguards. In this work, we examine the vulnerability of contemporary language models to automated, adversarial prompt refinement. We repurpose black-box prompt optimization techniques, originally designed to improve performance on benign tasks, to systematically search for safety failures. Using DSPy…
Prior work it builds on
-
LLMs as Signal Detectors: Sensitivity, Bias, and the Temperature-Criterion AnalogyJon-Paul Cacioli · 2026 · arXiv.org · preprint, not peer reviewed · cited 9 times · Not summarized yet
The study applies signal detection theory to separate sensitivity from response bias in factual judgments, providing a methodological precedent for this paper's safety analysis.
Abstract
Large language models (LLMs) are evaluated for calibration using metrics such as Expected Calibration Error that conflate two distinct components: the model's ability to discriminate correct from incorrect answers (sensitivity) and its tendency toward confident or cautious responding (bias). Signal Detection Theory (SDT) decomposes these components. While SDT-derived metrics such as AUROC are increasingly used, the full parametric framework - unequal-variance model fitting, criterion estimation, z-ROC analysis - has not been applied to LLMs as signal detectors. In this pre-registered study, we treat three LLMs as observers performing factual discrimination across 168,000 trials and test whet…
-
G-Eval: NLG Evaluation using GPT-4 with Better Human AlignmentYang Liu, Dan Iter, Yi-Chong Xu et al. · 2023 · Conference on Empirical Methods in Natural Language Processing · cited 3,690 times · AI summary available
G-Eval presents an LLM-based evaluation framework, while this paper also uses an LLM judging pipeline to classify model responses.
Abstract
The quality of texts generated by natural language generation (NLG) systems is hard to measure automatically. Conventional reference-based metrics, such as BLEU and ROUGE, have been shown to have relatively low correlation with human judgments, especially for tasks that require creativity and diversity. Recent studies suggest using large language models (LLMs) as reference-free metrics for NLG evaluation, which have the benefit of being applicable to new tasks that lack human references. However, these LLM-based evaluators still have lower human correspondence than medium-size neural evaluators. In this work, we present G-Eval, a framework of using large language models with chain-of-thought…
Go further
How are these papers found?
We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.
Updated on 16 November 2026.