Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning

Classés selon la proximité de leur titre et de leur résumé avec ceux de cet article. Sélectionnez-en pour les lire ensemble.

Les articles les plus proches

1. Très proche Pas pertinent

   Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning

   Chong-Yu Fan, Jiancheng Liu, Li-Cong Lin et al. · 2024 · Neural Information Processing Systems · cité 203 fois · Pas encore résumé

   Résumé scientifique

   This work studies the problem of large language model (LLM) unlearning, aiming to remove unwanted data influences (e.g., copyrighted or harmful content) while preserving model utility. Despite the increasing demand for unlearning, a technically-grounded optimization framework is lacking. Gradient ascent (GA)-type methods, though widely used, are suboptimal as they reverse the learning process without controlling optimization divergence (i.e., deviation from the pre-trained state), leading to risks of over-forgetting and potential model collapse. Negative preference optimization (NPO) has been proposed to address this issue and is considered one of the state-of-the-art LLM unlearning approach…

2. Très proche Cite cet article Utilise cet article en profondeur Pas pertinent

   RADNPO: Reference-free Adaptive Negative Preference Optimization for LLM Unlearning

   Sheng-Han Tan, Zi-Yi Zhou, Wen-Peng Hu et al. · 2026 · Pas encore résumé

   Résumé scientifique

   Large language models (LLMs) can memorize sensitive, private, or copyrighted content during pre-training, making machine unlearning necessary for removing targeted knowledge. Recent preference optimization (PO)-based unlearning methods improve stability over gradient ascent (GA)-based methods by introducing alignment-style objectives, which effectively suppress the probability of forget targets. However, target suppression alone does not sufficiently constrain the next-token distribution after unlearning. Existing methods provide limited control over how suppressed probability mass is redistributed and insufficiently adapt forgetting strength to target confidence and distributional concentra…

3. Très proche Pas pertinent

   Distribution Preference Optimization: A Fine-grained Perspective for LLM Unlearning

   Kai Qin, Jia-Qi Wu, Jianxiang He et al. · 2025 · arXiv.org · preprint, non relu par les pairs · cité 5 fois · Pas encore résumé

   Résumé scientifique

   As Large Language Models (LLMs) demonstrate remarkable capabilities learned from vast corpora, concerns regarding data privacy and safety are receiving increasing attention. LLM unlearning, which aims to remove the influence of specific data while preserving overall model utility, is becoming an important research area. One of the mainstream unlearning classes is optimization-based methods, which achieve forgetting directly through fine-tuning, exemplified by Negative Preference Optimization (NPO). However, NPO's effectiveness is limited by its inherent lack of explicit positive preference signals. Attempts to introduce such signals by constructing preferred responses often necessitate domai…

4. Très proche Pas pertinent

   Alternate Preference Optimization for Unlearning Factual Knowledge in Large Language Models

   Anmol Reddy Mekala, Vineeth Dorna, Shreya Dubey et al. · 2024 · arXiv.org · preprint, non relu par les pairs · cité 39 fois · Pas encore résumé

   Résumé scientifique

   Machine unlearning aims to efficiently eliminate the influence of specific training data, known as the forget set, from the model. However, existing unlearning methods for Large Language Models (LLMs) face a critical challenge: they rely solely on negative feedback to suppress responses related to the forget set, which often results in nonsensical or inconsistent outputs, diminishing model utility and posing potential privacy risks. To address this limitation, we propose a novel approach called Alternate Preference Optimization (AltPO), which combines negative feedback with in-domain positive feedback on the forget set. Additionally, we introduce new evaluation metrics to assess the quality …

5. Très proche Pas pertinent

   Multi-Objective Large Language Model Unlearning

   Zi-Bin Pan, Shu-Wen Zhang, Yuesheng Zheng et al. · 2024 · IEEE International Conference on Acoustics, Speech, and Signal Processing · cité 15 fois · Pas encore résumé

   Résumé scientifique

   Machine unlearning in the domain of large language models (LLMs) has attracted great attention recently, which aims to effectively eliminate undesirable behaviors from LLMs without full retraining from scratch. In this paper, we explore the Gradient Ascent (GA) approach in LLM unlearning, which is a proactive way to decrease the prediction probability of the model on the target data in order to remove their influence. We analyze two challenges that render the process impractical: gradient explosion and catastrophic forgetting. To address these issues, we propose Multi-Objective Large Language Model Unlearning (MOLLM) algorithm. We first formulate LLM unlearning as a multi-objective optimizat…

6. Très proche Cite cet article Pas pertinent

   Randomized Antipodal Search Done Right for Data Pareto Improvement of LLM Unlearning

   Zi-Wen Liu, Huawei Lin, Yi-De Ran et al. · 2026 · International Conference on Learning Representations · cité 2 fois · Pas encore résumé

   Résumé scientifique

   Large language models (LLMs) sometimes memorize undesirable knowledge, which must be removed after deployment. Prior work on machine unlearning has focused largely on optimization methods that adjust parameters to enforce forgetting while preserving retention. However, these approaches assume that the forget and retain sets are readily available, which rarely holds in practice. Unlearning is typically triggered by an undesired generation at inference time, making the retrieval of relevant data the central challenge. We introduce the notion of data Pareto improvement for LLM unlearning, which formalizes how retrieval can expand the achievable trade-off frontier between forgetting and retentio…

7. Très proche Cite cet article Pas pertinent

   Understanding Negative Learning in Large Language Model Fine-tuning

   Yue Wang, Qi-Zhou Wang, Zi-Zhuo Zhang et al. · 2025 · cité 4 fois · Pas encore résumé

   Résumé scientifique

   Negative learning has never been more important than it is today, particularly in large language model (LLM) fine-tuning, where it underpins machine unlearning, preference- and reward-based optimization, and exploratory data generation. Yet its functional role remains a subject of ongoing debate, with existing work variously characterizing it as forgetting, regularization, or target shaping. However, analytical support focuses primarily on the single-update scale, while empirical results at the trajectory scale are typically obtained under negative-only training and may not extend to composite objectives in which it interacts with other components. To develop a deeper understanding, we intro…

8. Très proche Pas pertinent

   Leveraging Machine Unlearning for Cost-Efficient Preference Alignment

   Xiaohua Feng, Yuyuan Li, Hu-Wei Ji et al. · 2025 · cité 3 fois · Pas encore résumé

   Résumé scientifique

   Despite advances in Preference Alignment (PA) for Large Language Models (LLMs), mainstream methods like reinforcement learning with human feedback face notable challenges. These approaches require high-quality datasets of positive preference examples, which are costly to obtain and computationally intensive. The LLM unlearning technique presents a promising alternative by directly removing the influence of negative examples. However, current research has primarily focused on empirical validation, lacking systematic quantitative analysis. To bridge this gap, we propose a framework linking PA with LLM unlearning. Through bi-level optimization, we first quantify how unlearning specific negative…

Les travaux antérieurs sur lesquels il s’appuie

1. Proche Utilisé en profondeur par cet article Une méthode qu’il utilise Pas pertinent

   Unlearn What You Want to Forget: Efficient Unlearning for LLMs

   Jiaao Chen, Di-Yi Yang · 2023 · Conference on Empirical Methods in Natural Language Processing · cité 283 fois · Pas encore résumé

   Résumé scientifique

   Large language models (LLMs) have achieved significant progress from pre-training on and memorizing a wide range of textual data, however, this process might suffer from privacy issues and violations of data protection regulations. As a result, the ability to easily remove data related to individual users from such models while not deteriorating their predictive quality after the removal becomes increasingly important. To address these issues, in this work, we propose an efficient unlearning framework that could efficiently update LLMs without having to retrain the whole model after data removals, by introducing lightweight unlearning layers learned with a selective teacher-student objective…

2. Proche Utilisé en profondeur par cet article Une méthode qu’il utilise Pas pertinent

   TOFU: A Task of Fictitious Unlearning for LLMs

   Pratyush Maini, Zhili Feng, Avi Schwarzschild et al. · 2024 · Machine Learning · preprint, non relu par les pairs · cité 659 fois · Résumé IA disponible

   Résumé scientifique

   Large language models trained on massive corpora of data from the web can memorize and reproduce sensitive or private data raising both legal and ethical concerns. Unlearning, or tuning models to forget information present in their training data, provides us with a way to protect private data after training. Although several methods exist for such unlearning, it is unclear to what extent they result in models equivalent to those where the data to be forgotten was never learned in the first place. To address this challenge, we present TOFU, a Task of Fictitious Unlearning, as a benchmark aimed at helping deepen our understanding of unlearning. We offer a dataset of 200 diverse synthetic autho…

3. Proche Utilisé en profondeur par cet article Une méthode qu’il utilise Pas pertinent

   Knowledge Unlearning for Mitigating Privacy Risks in Language Models

   Joel Jang, Dongkeun Yoon, Sohee Yang et al. · 2022 · Annual Meeting of the Association for Computational Linguistics · cité 587 fois · Pas encore résumé

   Résumé scientifique

   Pretrained Language Models (LMs) memorize a vast amount of knowledge during initial pretraining, including information that may violate the privacy of personal lives and identities. Previous work addressing privacy issues for LMs has mostly focused on data preprocessing and differential privacy methods, both requiring re-training the underlying LM. We propose knowledge unlearning as an alternative method to reduce privacy risks for LMs post hoc. We show that simply performing gradient ascent on target token sequences is effective at forgetting them with little to no degradation of general language modeling performances for larger-sized LMs. We also find that sequential unlearning is better t…

4. Proche Utilisé en profondeur par cet article Une méthode qu’il utilise Pas pertinent

   Direct Preference Optimization: Your Language Model is Secretly a Reward Model

   Rafael Rafailov, Archit Sharma, Eric Mitchell et al. · 2023 · Machine Learning · cité 11 469 fois · Résumé IA disponible

   Résumé scientifique

   While large-scale unsupervised language models (LMs) learn broad world knowledge and some reasoning skills, achieving precise control of their behavior is difficult due to the completely unsupervised nature of their training. Existing methods for gaining such steerability collect human labels of the relative quality of model generations and fine-tune the unsupervised LM to align with these preferences, often with reinforcement learning from human feedback (RLHF). However, RLHF is a complex and often unstable procedure, first fitting a reward model that reflects the human preferences, and then fine-tuning the large unsupervised LM using reinforcement learning to maximize this estimated reward…

5. Proche Une méthode qu’il utilise Pas pertinent

   Towards Safer Large Language Models through Machine Unlearning

   Zheyuan Liu, Guangyao Dou, Zhaoxuan Tan et al. · 2024 · Annual Meeting of the Association for Computational Linguistics · cité 174 fois · Pas encore résumé

   Résumé scientifique

   The rapid advancement of Large Language Models (LLMs) has demonstrated their vast potential across various domains, attributed to their extensive pretraining knowledge and exceptional generalizability. However, LLMs often encounter challenges in generating harmful content when faced with problematic prompts. To address this problem, existing work attempted to implement a gradient ascent based approach to prevent LLMs from producing harmful output. While these methods can be effective, they frequently impact the model utility in responding to normal prompts. To address this gap, we introduce Selective Knowledge negation Unlearning (SKU), a novel unlearning framework for LLMs, designed to elim…

Les articles qui le citent

1. Très proche Utilise cet article en profondeur Pas pertinent

   Modeling LLM Unlearning as an Asymmetric Two-Task Learning Problem

   Zeguan Xiao, Si-Yu Li, Yong Wang et al. · 2026 · Annual Meeting of the Association for Computational Linguistics · cité 4 fois · Pas encore résumé

   Résumé scientifique

   Machine unlearning for large language models (LLMs) aims to remove targeted knowledge while preserving general capability. In this paper, we recast LLM unlearning as an asymmetric two-task problem: retention is the primary objective and forgetting is an auxiliary. From this perspective, we propose a retention-prioritized gradient synthesis framework that decouples task-specific gradient extraction from conflict-aware combination. Instantiating the framework, we adapt established PCGrad to resolve gradient conflicts, and introduce SAGO, a novel retention-prioritized gradient synthesis method. Theoretically, both variants ensure non-negative cosine similarity with the retain gradient, while SA…

2. Très proche Utilise cet article en profondeur Pas pertinent

   De-attribute to Forget for LLM Unlearning

   Xin-Yang Lu, Jiabao Pan, Rachael Hwee Ling Sim et al. · 2026 · arXiv.org · preprint, non relu par les pairs · cité 1 fois · Pas encore résumé

   Résumé scientifique

   The rapid development of large language models (LLMs) has raised concerns on the use of inappropriate data for training, which has led to a growing interest in LLM unlearning. Many existing LLM unlearning approaches rely on optimizing prediction loss(es), such as maximizing the loss on the forget set, but often face critical issues like over-forgetting and poor model utility. To address them, this paper novelly frames the optimization objective for LLM unlearning as one of zeroing out data attribution instead. In particular, we propose the first LLM unlearning framework based on data attribution rewards called DareU that performs reinforcement learning to update the LLM by reducing the attri…

3. Proche Utilise cet article en profondeur Pas pertinent

   BalDRO: A Distributionally Robust Optimization based Framework for Large Language Model Unlearning

   Pengyang Shao, Nai-Xin Zhai, Lei Chen et al. · 2026 · The Web Conference · cité 14 fois · Pas encore résumé

   Résumé scientifique

   As Large Language Models (LLMs) increasingly shape online content, how to remove targeted information from well-trained LLMs (also known as LLM unlearning) has become increasingly critical for web governance. A key challenge in LLM unlearning lies in the sample-wise imbalance within the forget set: different samples exhibit widely varying unlearning difficulty, leading to asynchronous forgetting speeds where some knowledge remains insufficiently erased while others become over-forgotten. To address this challenge, we propose BalDRO, a novel and efficient framework for balanced LLM unlearning. BalDRO formulates unlearning as a min–sup process, where the inner process identifies a worst-case d…

4. Proche Utilise cet article en profondeur Pas pertinent

   Stress Testing Unlearning Algorithms

   Noam Diamant, Ethan Fetaya, Neta Glazer · 2026 · arXiv.org · preprint, non relu par les pairs · cité 1 fois · Pas encore résumé

   Résumé scientifique

   Recently, machine unlearning, the removal of specific training data influence from a model, has gained increasing attention. In large language models (LLMs), unlearning is particularly challenging due to the ambiguity of inputs and outputs. Con- sequently, rigorous evaluation is critical for assessing both safety and utility, and for driving progress in unlearning meth- ods. We identify two key shortcomings in existing unlearning benchmarks: (1) they do not actively test whether unlearned information can still be forcibly extracted, and (2) they fail to evaluate performance preservation on boundary questions, be- nign queries that are semantically close to the unlearned con- tent. Here we in…

5. Proche Utilise cet article en profondeur Pas pertinent

   Preserving Utility in Continual LLM Unlearning via Orthogonal Projections

   Juan Belieni, Ana Carolina Erthal, Eliezer de Souza da Silva et al. · 2026 · Pas encore résumé

   Résumé scientifique

   Unlearning concerns the task of removing the influence of a sensitive portion of the training data, such as protected, copyrighted or private text, from a model without retraining it from scratch. While unlearning requests happen continuously during a model's lifetime, the vast majority of LLM unlearning methods are ill-suited for iterated removal, yielding steep decay in forgetting and utility upon sequential unlearning. This decay stems from interference between successive updates, as each new deletion can undo earlier ones or push the model further along directions already degraded by them. To address this, we introduce Continual Orthogonal Projection (COP), a gradient-projection scheme t…