Invisibilization as a Source of Probability Bias in Language Models

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Cited by this paper Not relevant

   Evaluating and Mitigating Discrimination in Language Model Decisions

   Alex Tamkin, Amanda Askell, Liane Lovitt et al. · 2023 · Computation and Language · AI summary available

   Abstract

   As language models (LMs) advance, interest is growing in applying them to high-stakes societal decisions, such as determining financing or housing eligibility. However, their potential for discrimination in such contexts raises ethical concerns, motivating the need for better methods to evaluate these risks. We present a method for proactively evaluating the potential discriminatory impact of LMs in a wide range of use cases, including hypothetical use cases where they have not yet been deployed. Specifically, we use an LM to generate a wide array of potential prompts that decision-makers may input into an LM, spanning 70 diverse decision scenarios across society, and systematically vary the…

2. Very close Not relevant

   Self-Blinding and Counterfactual Self-Simulation Mitigate Biases and Sycophancy in Large Language Models

   Brian Christian, Matan Mazor · 2026 · arXiv.org · Not summarized yet

   Abstract

   Fair decisions require ignoring irrelevant, potentially biasing, information. To achieve this, decision-makers need to approximate what decision they would have made had they not known certain facts, such as the gender or race of a job candidate. This counterfactual self-simulation is notoriously hard for humans, leading to biased judgments even by well-meaning actors. Here we show that large language models (LLMs) suffer from similar limitations in their ability to approximate what decisions they would make under counterfactual knowledge in offsetting gender and race biases and overcoming sycophancy. We show that prompting models to ignore or pretend not to know biasing information fails to…

3. Very close Not relevant

   OptimismBench: Forecasting Bias and the Alignment Effect in Language Model Judgment

   Seonglae Cho, A. Koshiyama · 2026 · arXiv.org · Not summarized yet

   Abstract

   Large language models are increasingly used as decision aids whose probability judgments shape downstream choices. Whether those judgments carry a systematic directional tilt has been hard to detect: calibration metrics aggregate unsigned errors, and naturalistic uncertainty offers no ground-truth probability. When an LLM rates a startup's success at 70% but its failure at 15%, the missing 15 points expose a distortion no aggregate score flags. We introduce OptimismBench, which detects directional bias with inverted pairs: each scenario elicits both P(success) and P(failure), and asymmetry between the two framings yields a signed bias score without ground truth. Across 16 models from 8 provi…

4. Close Not relevant

   Behind the Words: Unmasking Bias Lurking in LLMs and Data

   Naina Dhingra, Gargi Kale, Harshad Chavan et al. · 2025 · 2025 IEEE Pune Section International Conference (PuneCon) · Not summarized yet

   Abstract

   This study examines hidden biases in large language models (LLMs). Three popular models-RoBERTa, BERT, and GPT-2-were evaluated using StereoSet, CrowS-Pairs, and Indian-BhED, datasets designed to expose biases. Our comparative analysis reveals clear variations in how these models encode and reproduce bias, underscoring the difficulty of mitigation. The results highlight the need for bias-aware evaluation. By contributing empirical evidence, this work advances the discussion on ethical AI and points to directions for building more responsible language technologies.

5. Close Cited by this paper Not relevant

   Characterizing Selective Refusal Bias in Large Language Models

   Adel Khorramrouz, Sharon Levy · 2025 · Annual Meeting of the Association for Computational Linguistics · Not summarized yet

   Abstract

   Safety guardrails in large language models(LLMs) are developed to prevent malicious users from generating toxic content at a large scale. However, these measures can inadvertently introduce or reflect new biases, as LLMs may refuse to generate harmful content targeting some demographic groups and not others. We explore this selective refusal bias in LLM guardrails through the lens of refusal rates of targeted individual and intersectional demographic groups, types of LLM responses, and length of generated refusals. Our results show evidence of selective refusal bias across gender, sexual orientation, nationality, and religion attributes. This leads us to investigate additional safety implica…

6. Close Not relevant

   Moral Safety in LLMs: Exposing Performative Compliance with Puzzled Cues

   Mohammadamin Shafiei, S. Li, Y. Tsvetkov · 2026 · arXiv.org · Not summarized yet

   Abstract

   As large language models take on morally consequential roles in healthcare, legal, and hiring contexts, we need to examine whether their ethical behaviors are genuine or superficial. We show that current fairness evaluations substantially overestimate moral safety. Models appear fair when demographic identity is stated as an explicit label, yet become measurably less fair when the same identity must be inferred. We term this failure performative compliance, where a model is fair when the presentation resembles a fairness evaluation and less fair as that cue weakens. We introduce a cue-variation methodology that holds the moral dilemma and the demographic identity fixed and varies only how th…

7. Close Not relevant

   Value Leakage: An LLM's Answers Are Silently Shaped by Its Own Values

   Jan Betley, Johannes Treutlein, Jan Dubinski et al. · 2026 · arXiv.org · Not summarized yet

   Abstract

   People use language models for practical questions whose answers are difficult to verify. We show that models exhibit covert value leakage: the information they provide is influenced by their own values, without this influence being disclosed to the user. In one of our evaluations, the user is considering investing in an AI company and wants to know how likely the AI bubble is to pop. Claude Opus 4.8 gives a lower probability when the company under consideration is Anthropic rather than OpenAI. Yet Claude mostly fails to disclose this influence to the user. Covert value leakage is a form of misalignment because it goes against the user's preferences and is likely to mislead them. To investig…

8. Close Not relevant

   Technical Report: Large Language Models can Strategically Deceive their Users when Put Under Pressure

   Jérémy Scheurer, Mikita Balesni, Marius Hobbhahn · 2023 · Computation and Language · AI summary available

   Abstract

   We demonstrate a situation in which Large Language Models, trained to be helpful, harmless, and honest, can display misaligned behavior and strategically deceive their users about this behavior without being instructed to do so. Concretely, we deploy GPT-4 as an agent in a realistic, simulated environment, where it assumes the role of an autonomous stock trading agent. Within this environment, the model obtains an insider tip about a lucrative stock trade and acts upon it despite knowing that insider trading is disapproved of by company management. When reporting to its manager, the model consistently hides the genuine reasons behind its trading decision. We perform a brief investigation of …

Prior work it builds on

1. Close Not relevant

   The Refusal-Compliance Tradeoff: A Large-Scale Safety Behavior Audit of Large Language Models

   Alif Al Hasan, Sumon Biswas · 2026 · arXiv.org · Not summarized yet

   Abstract

   Refusal rates are a poor proxy for LLM safety, i.e., a model may over-refuse benign prompts while still complying with harmful ones. We audit both failure modes across 21 open-weight LLMs on four safety benchmarks (OR-Bench, XSTest, ToxiGen, BOLD), using a composition adjustment to isolate model sensitivity from dataset toxicity confounds. We report three findings. First, models adopt fundamentally different calibration strategies: conservative ecosystems such as Llama suppress unsafe outputs at the cost of elevated over-refusals, while permissive ecosystems such as DeepSeek and Qwen preserve helpfulness but tolerate higher harmful compliance. Second, demographic protection is unequal: model…

2. Close Not relevant

   Popular LLMs Amplify Race and Gender Disparities in Human Mobility

   Xinhua Wu, Qi Wang · 2024 · arXiv.org · Not summarized yet

   Abstract

   As large language models (LLMs) are increasingly applied in areas influencing societal outcomes, it is critical to understand their tendency to perpetuate and amplify biases. This study investigates whether LLMs exhibit biases in predicting human mobility -- a fundamental human behavior -- based on race and gender. Using three prominent LLMs -- GPT-4, Gemini, and Claude -- we analyzed their predictions of visitations to points of interest (POIs) for individuals, relying on prompts that included names with and without explicit demographic details. We find that LLMs frequently reflect and amplify existing societal biases. Specifically, predictions for minority groups were disproportionately sk…

3. Close Not relevant

   Training language models to be warm can reduce accuracy and increase sycophancy

   Lujain Ibrahim, Franziska Sofia Hafner, Luc Rocher · 2026 · Nature · Not summarized yet

   Abstract

   Artificial intelligence developers are increasingly building language models with warm and friendly personas that millions of people now use for advice, therapy and companionship1. Here we show how this can create a significant trade-off: optimizing language models for warmth can undermine their performance, especially when users express vulnerability. We conducted controlled experiments on five different language models, training them to produce warmer responses, then evaluating them on consequential tasks. Warm models showed substantially higher error rates (+10 to +30 percentage points) than their original counterparts, promoting conspiracy theories, providing inaccurate factual informati…

4. Close Not relevant

   Towards Understanding Sycophancy in Language Models

   Mrinank Sharma, Meg Tong, Tomasz Korbak et al. · 2023 · Computation and Language · AI summary available

   Abstract

   Human feedback is commonly utilized to finetune AI assistants. But human feedback may also encourage model responses that match user beliefs over truthful ones, a behaviour known as sycophancy. We investigate the prevalence of sycophancy in models whose finetuning procedure made use of human feedback, and the potential role of human preference judgments in such behavior. We first demonstrate that five state-of-the-art AI assistants consistently exhibit sycophancy across four varied free-form text-generation tasks. To understand if human preferences drive this broadly observed behavior, we analyze existing human preference data. We find that when a response matches a user's views, it is more …

5. Related Not relevant

   XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models

   Paul Röttger, Hannah Rose Kirk, Bertie Vidgen et al. · 2023 · North American Chapter of the Association for Computational Linguistics · Not summarized yet

   Abstract

   Without proper safeguards, large language models will readily follow malicious instructions and generate toxic content. This risk motivates safety efforts such as red-teaming and large-scale feedback learning, which aim to make models both helpful and harmless. However, there is a tension between these two objectives, since harmlessness requires models to refuse to comply with unsafe prompts, and thus not be helpful. Recent anecdotal evidence suggests that some models may have struck a poor balance, so that even clearly safe prompts are refused if they use similar language to unsafe prompts or mention sensitive topics. In this paper, we introduce a new test suite called XSTest to identify su…