The Development of Sycophancy in Language Models: Tracing Vulnerabilities from Pretraining to Post-Training

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Close Not relevant

   Towards Understanding Sycophancy in Language Models

   Mrinank Sharma, Meg Tong, Tomasz Korbak et al. · 2023 · Computation and Language · AI summary available

   Abstract

   Human feedback is commonly utilized to finetune AI assistants. But human feedback may also encourage model responses that match user beliefs over truthful ones, a behaviour known as sycophancy. We investigate the prevalence of sycophancy in models whose finetuning procedure made use of human feedback, and the potential role of human preference judgments in such behavior. We first demonstrate that five state-of-the-art AI assistants consistently exhibit sycophancy across four varied free-form text-generation tasks. To understand if human preferences drive this broadly observed behavior, we analyze existing human preference data. We find that when a response matches a user's views, it is more …

2. Close Not relevant

   Alignment faking in large language models

   Ryan Greenblatt, Carson Denison, Benjamin Wright et al. · 2024 · Artificial Intelligence · AI summary available

   Abstract

   We present a demonstration of a large language model engaging in alignment faking: selectively complying with its training objective in training to prevent modification of its behavior out of training. First, we give Claude 3 Opus a system prompt stating it is being trained to answer all queries, even harmful ones, which conflicts with its prior training to refuse such queries. To allow the model to infer when it is in training, we say it will be trained only on conversations with free users, not paid users. We find the model complies with harmful queries from free users 14% of the time, versus almost never for paid users. Explaining this gap, in almost all cases where the model complies wit…

3. Close Cited by this paper Not relevant

   Ask Again, Then Fail: Large Language Models’ Vacillations in Judgment

   Qiming Xie, Zengzhi Wang, Yihao Feng et al. · 2024 · Volume 1 · Not summarized yet

   Abstract

   We observe that current large language models often waver in their judgments when faced with follow-up questions, even if the original judgment was correct. This wavering presents a significant challenge for generating reliable responses and building user trust. To comprehensively assess this issue, we introduce a F OLLOW - UP Q UESTIONING M ECHANISM along with two metrics to quantify this inconsistency, confirming its widespread presence in current large language models. Furthermore, to mitigate this issue, we explore various prompting strategies for closed-source models, and develop a training-based framework U NWAVERING -FQ that teaches large language models to maintain their originally cor…

4. Close Cited by this paper Not relevant

   Measuring Sycophancy of Language Models in Multi-turn Dialogues

   Jiseung Hong, Grace Byun, Seungone Kim et al. · 2025 · Conference on Empirical Methods in Natural Language Processing · Not summarized yet

   Abstract

   Large Language Models (LLMs) are expected to provide helpful and harmless responses, yet they often exhibit sycophancy--conforming to user beliefs regardless of factual accuracy or ethical soundness. Prior research on sycophancy has primarily focused on single-turn factual correctness, overlooking the dynamics of real-world interactions. In this work, we introduce SYCON Bench, a novel benchmark for evaluating sycophantic behavior in multi-turn, free-form conversational settings. Our benchmark measures how quickly a model conforms to the user (Turn of Flip) and how frequently it shifts its stance under sustained user pressure (Number of Flip). Applying SYCON Bench to 17 LLMs across three real…

5. Close Not relevant

   Invisibilization as a Source of Probability Bias in Language Models

   Christian Arnold, Ernesto Calvo, Guadalupe Gonzalez et al. · 2026 · Computation and Language · AI summary available

   In short

   Leading language models often refuse to assess reports based on the reporter’s description, producing racial gaps in information use that average-trust audits can miss.

6. Close Not relevant

   Emergently Misaligned Language Models Show Behavioral Self-Awareness That Shifts With Subsequent Realignment

   Laurène Vaugrante, Anietta Weckauff, Thilo Hagendorff · 2026 · arXiv.org · Not summarized yet

   Abstract

   Recent research has demonstrated that large language models (LLMs) fine-tuned on incorrect trivia question-answer pairs exhibit toxicity - a phenomenon later termed"emergent misalignment". Moreover, research has shown that LLMs possess behavioral self-awareness - the ability to describe learned behaviors that were only implicitly demonstrated in training data. Here, we investigate the intersection of these phenomena. We fine-tune GPT-4.1 models sequentially on datasets known to induce and reverse emergent misalignment and evaluate whether the models are self-aware of their behavior transitions without providing in-context examples. Our results show that emergently misaligned models rate them…

7. Close Cited by this paper Not relevant

   How Does Alignment Tuning Shape Representations of Sycophancy and Related Cue-Induced Biases in LLMs?

   Prakhar Gupta, Terry Jingchen Zhang, Florent Draye et al. · 2026 · arXiv.org · Not summarized yet

   Abstract

   Modern LLMs are alarmingly susceptible to surprisingly simple immaterial changes of input prompts: a casual hint, an incorrectly labeled few-shot example, or a fake prior assistant turn often flips an originally correct answer. We study where this susceptibility, spanning sycophancy and related cue-induced biases, lives inside the model. Across five model families and seven bias types, we extract a per-bias direction from hidden states and triangulate it through three measures: probing, leave-one-dataset-out (LODO) transfer, and causal intervention. The susceptibility is largely shaped by alignment tuning rather than pretraining: pretrained base models generally cave much less to these biase…

8. Close Not relevant

   Language Models (Mostly) Know What They Know

   Saurav Kadavath, Tom Conerly, Amanda Askell et al. · 2022 · Computation and Language · AI summary available

   Abstract

   We study whether language models can evaluate the validity of their own claims and predict which questions they will be able to answer correctly. We first show that larger models are well-calibrated on diverse multiple choice and true/false questions when they are provided in the right format. Thus we can approach self-evaluation on open-ended sampling tasks by asking models to first propose answers, and then to evaluate the probability "P(True)" that their answers are correct. We find encouraging performance, calibration, and scaling for P(True) on a diverse array of tasks. Performance at self-evaluation further improves when we allow models to consider many of their own samples before pred…

Prior work it builds on

1. Close Not relevant

   Discovering Language Model Behaviors with Model-Written Evaluations

   Ethan Perez, Sam Ringer, Kamilė Lukošiūtė et al. · 2022 · Computation and Language · AI summary available

   Abstract

   As language models (LMs) scale, they develop many novel behaviors, good and bad, exacerbating the need to evaluate how they behave. Prior work creates evaluations with crowdwork (which is time-consuming and expensive) or existing data sources (which are not always available). Here, we automatically generate evaluations with LMs. We explore approaches with varying amounts of human effort, from instructing LMs to write yes/no questions to making complex Winogender schemas with multiple stages of LM-based generation and filtering. Crowdworkers rate the examples as highly relevant and agree with 90-100% of labels, sometimes more so than corresponding human-written datasets. We generate 154 datas…

2. Close Not relevant

   TruthfulQA: Measuring How Models Mimic Human Falsehoods

   Stephanie Lin, Jacob Hilton, Owain Evans · 2021 · Computation and Language · AI summary available

   Abstract

   We propose a benchmark to measure whether a language model is truthful in generating answers to questions. The benchmark comprises 817 questions that span 38 categories, including health, law, finance and politics. We crafted questions that some humans would answer falsely due to a false belief or misconception. To perform well, models must avoid generating false answers learned from imitating human texts. We tested GPT-3, GPT-Neo/J, GPT-2 and a T5-based model. The best model was truthful on 58% of questions, while human performance was 94%. Models generated many false answers that mimic popular misconceptions and have the potential to deceive humans. The largest models were generally the le…

3. Close Not relevant

   Sycophantic Agreement Transfers with Neutral Data via Contrastive Preference Optimization

   C. Blank, Zhuo-Fan Ying, Christopher Potts et al. · 2026 · Not summarized yet

   Abstract

   Sycophantic agreement refers to a behavior in which language models excessively affirm the user, often at the cost of factual accuracy. Although sycophantic agreement is a well-known failure of model alignment, there is limited understanding of how it emerges from model training. In this work, we demonstrate that sycophantic agreement can emerge as an unintended consequence of widely used contrastive preference optimization objectives. Using the OLMo 3 post-training pipeline, we show that, for various pairs of teacher models across three families, there is a strong correlation between the log-ratio of the teacher model sycophantic agreement rates and the resulting student model sycophantic a…

4. Related Not relevant

   Aligning AI With Shared Human Values

   Dan Hendrycks, Collin Burns, Steven Basart et al. · 2020 · International Conference on Learning Representations · Not summarized yet

   Abstract

   We show how to assess a language model's knowledge of basic concepts of morality. We introduce the ETHICS dataset, a new benchmark that spans concepts in justice, well-being, duties, virtues, and commonsense morality. Models predict widespread moral judgments about diverse text scenarios. This requires connecting physical and social world knowledge to value judgements, a capability that may enable us to steer chatbot outputs or eventually regularize open-ended reinforcement learning agents. With the ETHICS dataset, we find that current language models have a promising but incomplete understanding of basic ethical knowledge. Our work shows that progress can be made on machine ethics today, an…

5. Related Not relevant

   Anatomy of LLM Sycophancy: What a Flip Rate Hides

   Hao-Nan Huang · 2026 · Not summarized yet

   Abstract

   A model under pushback can correct itself, capitulate, or hold, and one flip rate counts a correction and a capitulation alike. Using SycoLens, a modular replay protocol, we test how user pressure and evaluation settings shape measured flip rates. Each measurement is one stateless replay of an item, a committed answer, and one scripted user line in a fixed form. Every effect is read against a matched control with the line deleted. Pushback wording, committed text, answer format, boundary distance, and ground truth become factors of one instrument; earlier instruments vary one to three of them. Across eleven frontier models from three providers and about 760,000 controlled replays, which mode…