AI summary · arXiv: 2611.11074v1 · cs.LG
System Prompts Affect Response Bias More Than Discriminability in LLM Safety
The essentials
Across 13 models and three safety benchmarks, system prompts more consistently changed refusal willingness and response policy than refusal-based discrimination between safe and unsafe requests.
Question
When the same safety-relevant request is presented under different system prompts, do prompts improve models’ ability to distinguish safe from unsafe requests, or mainly alter their willingness to refuse? The study also asks whether prompts route responses among behavioral policies beyond refusal and compliance.
Method
The authors crossed eight safety-task prompt families with three clarity levels and three variants per condition, evaluating 13 instruction-tuned models on items from HarmBench, XSTest, and IHEval while holding queries fixed within models. They used signal detection theory to estimate refusal-based discriminability and response criterion, and an output-only LLM judging pipeline to classify responses into six policies; the full-coverage evaluation used 3,370 selected benchmark items per prompt configuration, while frontier models used smaller subsets for the refusal analyses.
Findings
-
Result
Model identity strongly predicted both refusal-based discriminability and response criterion.
See the passage in the paper
Model identity strongly predicted discriminability, $F(12,290)=258.0$, $p<.001$, partial $\eta_{p}^{2}=.914$, indicating substantial differences in how effectively models distinguish safe from unsafe requests through refusal behavior.
-
Result
Prompt family had a larger and more consistent effect on refusal criterion than on discriminability.
See the passage in the paper
Prompt family had a much larger effect on response criterion, $F(7,290)=16.61$, $p<.001$, partial $\eta_{p}^{2}=.286$.
-
Result
Increasing prompt clarity lowered the refusal criterion but did not significantly improve discriminability.
See the passage in the paper
In contrast, clarity did not significantly affect discriminability, $F(2,290)=2.01$, $p=.136$, partial $\eta_{p}^{2}=.014$.
-
Result
Prompt families increased the frequency of their corresponding target policies, but did not determine responses reliably.
See the passage in the paper
Thus, prompt families systematically biased models toward their intended behavioral policies, but did not deterministically select them.
-
Authors’ interpretation
Clarity’s effects on the broader policy distribution varied with the prompt family.
See the passage in the paper
Instead, greater clarity changed the balance of available policies in ways that depended on the task being emphasized.
Limitations
- The output-only judging approach cannot establish whether a response correctly handled the original request or instruction conflict.
See the passage in the paper
Output-only labels also cannot establish whether a response is correct when the original request or instruction context is needed.
- The frontier models were tested on smaller request samples than the ten open-weight models, limiting direct comparison of absolute discriminability estimates.
See the passage in the paper
The different item sets therefore limit direct comparisons of absolute $d^{\prime}$ across models.
- The prompt families isolate individual safety priorities, so how the findings transfer to deployed systems combining multiple objectives and contextual sources remains uncertain.
See the passage in the paper
Whether our findings extend to these settings, including agentic workflows, remains an open question.
Why it matters
The authors argue that safety-prompt evaluation should distinguish changes in refusal willingness from changes in discrimination, and should measure response policies beyond the binary refuse-or-comply outcome. Their framework offers behavioral targets for prompt design and future mechanistic work, while leaving joint optimization of these dimensions as an open question.
Written by AI from the paper. Passages are checked word for word against the text we analysed; the rest may contain errors.
Ask questions about this paper to our AI assistant
How to chat with this paper? Illustrated guide
You can also chat with multiple papers at once here.
Keep this conversation — a free account saves your questions and answers, your collections of papers and your alerts.
Continue with Google Create a free accountTools for this paper
Advanced tools Premium
For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium
Papers around this one
See them on a mapRanked by how close their title and abstract are to this paper’s. Select some to read them together.
Closest papers
-
The Refusal-Compliance Tradeoff: A Large-Scale Safety Behavior Audit of Large Language ModelsAlif Al Hasan, Sumon Biswas · 2026 · arXiv.org · preprint, not peer reviewed · cited 1 time · Not summarized yet
The audit compares harmful compliance with benign over-refusal across safety benchmarks, examining the refusal-compliance balance that this paper separates into discriminability and refusal tendency.
Abstract
Refusal rates are a poor proxy for LLM safety, i.e., a model may over-refuse benign prompts while still complying with harmful ones. We audit both failure modes across 21 open-weight LLMs on four safety benchmarks (OR-Bench, XSTest, ToxiGen, BOLD), using a composition adjustment to isolate model sensitivity from dataset toxicity confounds. We report three findings. First, models adopt fundamentally different calibration strategies: conservative ecosystems such as Llama suppress unsafe outputs at the cost of elevated over-refusals, while permissive ecosystems such as DeepSeek and Qwen preserve helpfulness but tolerate higher harmful compliance. Second, demographic protection is unequal: model…
-
Mitigating Exaggerated Safety in Large Language ModelsRuchira Ray, Ruchi Bhalani · 2024 · arXiv.org · preprint, not peer reviewed · cited 10 times · Not summarized yet
Mitigating exaggerated refusals uses interactive, contextual, and few-shot prompting on XSTest to reduce refusals of safe requests, rather than measuring system-prompt effects with signal detection theory.
Abstract
As the popularity of Large Language Models (LLMs) grow, combining model safety with utility becomes increasingly important. The challenge is making sure that LLMs can recognize and decline dangerous prompts without sacrificing their ability to be helpful. The problem of"exaggerated safety"demonstrates how difficult this can be. To reduce excessive safety behaviours -- which was discovered to be 26.1% of safe prompts being misclassified as dangerous and refused -- we use a combination of XSTest dataset prompts as well as interactive, contextual, and few-shot prompting to examine the decision bounds of LLMs such as Llama2, Gemma Command R+, and Phi-3. We find that few-shot prompting works best…
-
From Prompt Risk to Response Risk: Paired Analysis of Safety Behavior of Large Language ModelMeng-Ya Hu, Qiong Wei, Sandeep Atluri · 2026 · arXiv.org · preprint, not peer reviewed · Not summarized yet
The paired analysis tracks how model responses change harm severity relative to prompts, addressing safety behavior beyond the refusal-versus-compliance outcomes studied here.
Abstract
Safety evaluations of large language models (LLMs) typically report binary outcomes, i.e. attack success rate (ASR), refusal rate, or harmful versus safe classification, which hide how risk changes between prompt and response. We present a paired analysis over human labeled prompt and response records across four harm categories (Sexual, Self harm, Hate and Violence) and ordinal severity levels (Safe, Low, Medium, High). 61% of responses reduce harm relative to the prompt, 36% preserve severity, and 3% escalate. The escalation splits into two mechanisms: benign prompts triggering unrequested harmful detail, and answers that stay on task at higher severity than the prompt. Category decomposit…
-
Resisting Correction: How RLHF Makes Language Models Ignore External Safety Signals in Natural ConversationFelipe Biava Cataneo · 2025 · arXiv.org · preprint, not peer reviewed · cited 2 times · Not summarized yet
The intervention study tests whether models follow external confidence signals under different prompt strategies, but studies correction in math responses rather than safety-request discrimination.
Abstract
Safety architectures for language models increasingly rely on external monitors to detect errors and inject corrective signals at inference time. For such systems to function in interactive settings, models must be able to incorporate externally provided confidence information into their verbal responses. In this work, we test whether instruction-tuned language models preserve this controllability across different interaction modes. Using Llama-3.2-3B on GSM8K, we perform a causal intervention study in which explicit external confidence signals are injected and model compliance is measured under multiple prompt strategies. We find that base models exhibit near-perfect controllability (Spearm…
-
The Instability of Safety: How Random Seeds and Temperature Expose Inconsistent LLM Refusal BehaviorE. Larsen · 2025 · arXiv.org · preprint, not peer reviewed · cited 7 times · Not summarized yet
The study measures how sampling seeds and temperature change refusal decisions on harmful prompts, whereas this paper tests how system prompts shift safety behavior.
Abstract
Current safety evaluations of large language models rely on single-shot testing, implicitly assuming that model responses are deterministic and representative of the model's safety alignment. We challenge this assumption by investigating the stability of safety refusal decisions across random seeds and temperature settings. Testing four instruction-tuned models from three families (Llama 3.1 8B, Qwen 2.5 7B, Qwen 3 8B, Gemma 3 12B) on 876 harmful prompts across 20 sampling configurations (4 temperatures x 5 seeds), we find that 18-28% of prompts exhibit decision flips (the REFUSE/PARTIAL/COMPLY label changes across configurations), depending on the model. Our Safety Stability Index (SSI) rev…
-
Are Large Language Models Good Prompt Optimizers?Ruotian Ma, Xiaolei Wang, Xin Zhou et al. · 2024 · Computation and Language · preprint, not peer reviewed · cited 46 times · AI summary available
The study investigates how LLMs optimize prompts for task performance, rather than how system prompts alter safety responses.
Abstract
LLM-based Automatic Prompt Optimization, which typically utilizes LLMs as Prompt Optimizers to self-reflect and refine prompts, has shown promising performance in recent studies. Despite the success, the underlying mechanism of this approach remains unexplored, and the true effectiveness of LLMs as Prompt Optimizers requires further validation. In this work, we conducted a comprehensive study to uncover the actual mechanism of LLM-based Prompt Optimization. Our findings reveal that the LLM optimizers struggle to identify the true causes of errors during reflection, tending to be biased by their own prior knowledge rather than genuinely reflecting on the errors. Furthermore, even when the ref…
-
Beyond Over-Refusal: Scenario-Based Diagnostics and Post-Hoc Mitigation for Exaggerated Refusals in LLMsShu-Zhou Yuan, Ercong Nie, Yin Sun et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 5 times · Not summarized yet
The work diagnoses and mitigates false refusals with single- and multi-turn benchmarks, focusing on exaggerated refusal rather than prompt effects on discriminability.
Abstract
Large language models (LLMs) frequently produce false refusals, declining benign requests that contain terms resembling unsafe queries. We address this challenge by introducing two comprehensive benchmarks: the Exaggerated Safety Benchmark (XSB) for single-turn prompts, annotated with"Focus"keywords that identify refusal-inducing triggers, and the Multi-turn Scenario-based Exaggerated Safety Benchmark (MS-XSB), which systematically evaluates refusal calibration in realistic, context-rich dialog settings. Our benchmarks reveal that exaggerated refusals persist across diverse recent LLMs and are especially pronounced in complex, multi-turn scenarios. To mitigate these failures, we leverage pos…
-
When Prompt Optimization Becomes Jailbreaking: Adaptive Red-Teaming of Large Language ModelsZafir Shamsi, Nikhil Chekuru, Z. Guzman et al. · 2026 · Conference of the European Chapter of the Association for Computational Linguistics · cited 1 time · Not summarized yet
The work adaptively refines adversarial prompts to find safety failures, rather than comparing system prompts on fixed safety requests.
Abstract
Large Language Models (LLMs) are increasingly integrated into high-stakes applications, making robust safety guarantees a central practical and commercial concern. Existing safety evaluations predominantly rely on fixed collections of harmful prompts, implicitly assuming non-adaptive adversaries and thereby overlooking realistic attack scenarios in which inputs are iteratively refined to evade safeguards. In this work, we examine the vulnerability of contemporary language models to automated, adversarial prompt refinement. We repurpose black-box prompt optimization techniques, originally designed to improve performance on benign tasks, to systematically search for safety failures. Using DSPy…
Prior work it builds on
-
LLMs as Signal Detectors: Sensitivity, Bias, and the Temperature-Criterion AnalogyJon-Paul Cacioli · 2026 · arXiv.org · preprint, not peer reviewed · cited 9 times · Not summarized yet
The study applies signal detection theory to separate sensitivity from response bias in factual judgments, providing a methodological precedent for this paper's safety analysis.
Abstract
Large language models (LLMs) are evaluated for calibration using metrics such as Expected Calibration Error that conflate two distinct components: the model's ability to discriminate correct from incorrect answers (sensitivity) and its tendency toward confident or cautious responding (bias). Signal Detection Theory (SDT) decomposes these components. While SDT-derived metrics such as AUROC are increasingly used, the full parametric framework - unequal-variance model fitting, criterion estimation, z-ROC analysis - has not been applied to LLMs as signal detectors. In this pre-registered study, we treat three LLMs as observers performing factual discrimination across 168,000 trials and test whet…
-
G-Eval: NLG Evaluation using GPT-4 with Better Human AlignmentYang Liu, Dan Iter, Yi-Chong Xu et al. · 2023 · Conference on Empirical Methods in Natural Language Processing · cited 3,690 times · AI summary available
G-Eval presents an LLM-based evaluation framework, while this paper also uses an LLM judging pipeline to classify model responses.
Abstract
The quality of texts generated by natural language generation (NLG) systems is hard to measure automatically. Conventional reference-based metrics, such as BLEU and ROUGE, have been shown to have relatively low correlation with human judgments, especially for tasks that require creativity and diversity. Recent studies suggest using large language models (LLMs) as reference-free metrics for NLG evaluation, which have the benefit of being applicable to new tasks that lack human references. However, these LLM-based evaluators still have lower human correspondence than medium-size neural evaluators. In this work, we present G-Eval, a framework of using large language models with chain-of-thought…
Go further
How are these papers found?
We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.
Updated on 16 November 2026.
Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.