The Development of Sycophancy in Language Models: Tracing Vulnerabilities from Pretraining to Post-Training

Classés selon la proximité de leur titre et de leur résumé avec ceux de cet article. Sélectionnez-en pour les lire ensemble.

Les articles les plus proches

1. Proche Pas pertinent

   Towards Understanding Sycophancy in Language Models

   Mrinank Sharma, Meg Tong, Tomasz Korbak et al. · 2023 · Computation and Language · Résumé IA disponible

   Résumé scientifique

   Human feedback is commonly utilized to finetune AI assistants. But human feedback may also encourage model responses that match user beliefs over truthful ones, a behaviour known as sycophancy. We investigate the prevalence of sycophancy in models whose finetuning procedure made use of human feedback, and the potential role of human preference judgments in such behavior. We first demonstrate that five state-of-the-art AI assistants consistently exhibit sycophancy across four varied free-form text-generation tasks. To understand if human preferences drive this broadly observed behavior, we analyze existing human preference data. We find that when a response matches a user's views, it is more …

2. Proche Pas pertinent

   Alignment faking in large language models

   Ryan Greenblatt, Carson Denison, Benjamin Wright et al. · 2024 · Artificial Intelligence · Résumé IA disponible

   Résumé scientifique

   We present a demonstration of a large language model engaging in alignment faking: selectively complying with its training objective in training to prevent modification of its behavior out of training. First, we give Claude 3 Opus a system prompt stating it is being trained to answer all queries, even harmful ones, which conflicts with its prior training to refuse such queries. To allow the model to infer when it is in training, we say it will be trained only on conversations with free users, not paid users. We find the model complies with harmful queries from free users 14% of the time, versus almost never for paid users. Explaining this gap, in almost all cases where the model complies wit…

3. Proche Cité par cet article Pas pertinent

   Ask Again, Then Fail: Large Language Models’ Vacillations in Judgment

   Qiming Xie, Zengzhi Wang, Yihao Feng et al. · 2024 · Volume 1 · Pas encore résumé

   Résumé scientifique

   We observe that current large language models often waver in their judgments when faced with follow-up questions, even if the original judgment was correct. This wavering presents a significant challenge for generating reliable responses and building user trust. To comprehensively assess this issue, we introduce a F OLLOW - UP Q UESTIONING M ECHANISM along with two metrics to quantify this inconsistency, confirming its widespread presence in current large language models. Furthermore, to mitigate this issue, we explore various prompting strategies for closed-source models, and develop a training-based framework U NWAVERING -FQ that teaches large language models to maintain their originally cor…

4. Proche Cité par cet article Pas pertinent

   Measuring Sycophancy of Language Models in Multi-turn Dialogues

   Jiseung Hong, Grace Byun, Seungone Kim et al. · 2025 · Conference on Empirical Methods in Natural Language Processing · Pas encore résumé

   Résumé scientifique

   Large Language Models (LLMs) are expected to provide helpful and harmless responses, yet they often exhibit sycophancy--conforming to user beliefs regardless of factual accuracy or ethical soundness. Prior research on sycophancy has primarily focused on single-turn factual correctness, overlooking the dynamics of real-world interactions. In this work, we introduce SYCON Bench, a novel benchmark for evaluating sycophantic behavior in multi-turn, free-form conversational settings. Our benchmark measures how quickly a model conforms to the user (Turn of Flip) and how frequently it shifts its stance under sustained user pressure (Number of Flip). Applying SYCON Bench to 17 LLMs across three real…

5. Proche Pas pertinent

   Invisibilization as a Source of Probability Bias in Language Models

   Christian Arnold, Ernesto Calvo, Guadalupe Gonzalez et al. · 2026 · Computation and Language · Résumé IA disponible

   Résumé scientifique

   Language models are aligned to avoid using protected attributes such as race, ethnicity, and gender to inform their responses. Therefore, they increasingly refuse or hedge when such attributes appear in a prompt. However, if a system must decide how much to learn from what people tell it, it is costly to refuse valuable information because of its source. In this article, we measure the cost of declining information in a two-urn game with an exact Bayesian answer. We present leading LLM models from two developers (OpenAI and Anthropic) with a prompt that requires them to decide how far to trust an unverifiable report from a participant whose description we randomize across ten configurations …

6. Proche Pas pertinent

   Emergently Misaligned Language Models Show Behavioral Self-Awareness That Shifts With Subsequent Realignment

   Laurène Vaugrante, Anietta Weckauff, Thilo Hagendorff · 2026 · arXiv.org · Pas encore résumé

   Résumé scientifique

   Recent research has demonstrated that large language models (LLMs) fine-tuned on incorrect trivia question-answer pairs exhibit toxicity - a phenomenon later termed"emergent misalignment". Moreover, research has shown that LLMs possess behavioral self-awareness - the ability to describe learned behaviors that were only implicitly demonstrated in training data. Here, we investigate the intersection of these phenomena. We fine-tune GPT-4.1 models sequentially on datasets known to induce and reverse emergent misalignment and evaluate whether the models are self-aware of their behavior transitions without providing in-context examples. Our results show that emergently misaligned models rate them…

7. Proche Cité par cet article Pas pertinent

   How Does Alignment Tuning Shape Representations of Sycophancy and Related Cue-Induced Biases in LLMs?

   Prakhar Gupta, Terry Jingchen Zhang, Florent Draye et al. · 2026 · arXiv.org · Pas encore résumé

   Résumé scientifique

   Modern LLMs are alarmingly susceptible to surprisingly simple immaterial changes of input prompts: a casual hint, an incorrectly labeled few-shot example, or a fake prior assistant turn often flips an originally correct answer. We study where this susceptibility, spanning sycophancy and related cue-induced biases, lives inside the model. Across five model families and seven bias types, we extract a per-bias direction from hidden states and triangulate it through three measures: probing, leave-one-dataset-out (LODO) transfer, and causal intervention. The susceptibility is largely shaped by alignment tuning rather than pretraining: pretrained base models generally cave much less to these biase…

8. Proche Pas pertinent

   Language Models (Mostly) Know What They Know

   Saurav Kadavath, Tom Conerly, Amanda Askell et al. · 2022 · Computation and Language · Résumé IA disponible

   Résumé scientifique

   We study whether language models can evaluate the validity of their own claims and predict which questions they will be able to answer correctly. We first show that larger models are well-calibrated on diverse multiple choice and true/false questions when they are provided in the right format. Thus we can approach self-evaluation on open-ended sampling tasks by asking models to first propose answers, and then to evaluate the probability "P(True)" that their answers are correct. We find encouraging performance, calibration, and scaling for P(True) on a diverse array of tasks. Performance at self-evaluation further improves when we allow models to consider many of their own samples before pred…

Les travaux antérieurs sur lesquels il s’appuie

1. Proche Pas pertinent

   Discovering Language Model Behaviors with Model-Written Evaluations

   Ethan Perez, Sam Ringer, Kamilė Lukošiūtė et al. · 2022 · Computation and Language · Résumé IA disponible

   Résumé scientifique

   As language models (LMs) scale, they develop many novel behaviors, good and bad, exacerbating the need to evaluate how they behave. Prior work creates evaluations with crowdwork (which is time-consuming and expensive) or existing data sources (which are not always available). Here, we automatically generate evaluations with LMs. We explore approaches with varying amounts of human effort, from instructing LMs to write yes/no questions to making complex Winogender schemas with multiple stages of LM-based generation and filtering. Crowdworkers rate the examples as highly relevant and agree with 90-100% of labels, sometimes more so than corresponding human-written datasets. We generate 154 datas…

2. Proche Pas pertinent

   TruthfulQA: Measuring How Models Mimic Human Falsehoods

   Stephanie Lin, Jacob Hilton, Owain Evans · 2021 · Computation and Language · Résumé IA disponible

   Résumé scientifique

   We propose a benchmark to measure whether a language model is truthful in generating answers to questions. The benchmark comprises 817 questions that span 38 categories, including health, law, finance and politics. We crafted questions that some humans would answer falsely due to a false belief or misconception. To perform well, models must avoid generating false answers learned from imitating human texts. We tested GPT-3, GPT-Neo/J, GPT-2 and a T5-based model. The best model was truthful on 58% of questions, while human performance was 94%. Models generated many false answers that mimic popular misconceptions and have the potential to deceive humans. The largest models were generally the le…

3. Proche Pas pertinent

   Sycophantic Agreement Transfers with Neutral Data via Contrastive Preference Optimization

   C. Blank, Zhuo-Fan Ying, Christopher Potts et al. · 2026 · Pas encore résumé

   Résumé scientifique

   Sycophantic agreement refers to a behavior in which language models excessively affirm the user, often at the cost of factual accuracy. Although sycophantic agreement is a well-known failure of model alignment, there is limited understanding of how it emerges from model training. In this work, we demonstrate that sycophantic agreement can emerge as an unintended consequence of widely used contrastive preference optimization objectives. Using the OLMo 3 post-training pipeline, we show that, for various pairs of teacher models across three families, there is a strong correlation between the log-ratio of the teacher model sycophantic agreement rates and the resulting student model sycophantic a…

4. Apparenté Pas pertinent

   Aligning AI With Shared Human Values

   Dan Hendrycks, Collin Burns, Steven Basart et al. · 2020 · International Conference on Learning Representations · Pas encore résumé

   Résumé scientifique

   We show how to assess a language model's knowledge of basic concepts of morality. We introduce the ETHICS dataset, a new benchmark that spans concepts in justice, well-being, duties, virtues, and commonsense morality. Models predict widespread moral judgments about diverse text scenarios. This requires connecting physical and social world knowledge to value judgements, a capability that may enable us to steer chatbot outputs or eventually regularize open-ended reinforcement learning agents. With the ETHICS dataset, we find that current language models have a promising but incomplete understanding of basic ethical knowledge. Our work shows that progress can be made on machine ethics today, an…

5. Apparenté Pas pertinent

   Anatomy of LLM Sycophancy: What a Flip Rate Hides

   Hao-Nan Huang · 2026 · Pas encore résumé

   Résumé scientifique

   A model under pushback can correct itself, capitulate, or hold, and one flip rate counts a correction and a capitulation alike. Using SycoLens, a modular replay protocol, we test how user pressure and evaluation settings shape measured flip rates. Each measurement is one stateless replay of an item, a committed answer, and one scripted user line in a fixed form. Every effect is read against a matched control with the line deleted. Pushback wording, committed text, answer format, boundary distance, and ground truth become factors of one instrument; earlier instruments vary one to three of them. Across eleven frontier models from three providers and about 760,000 controlled replays, which mode…