Large-scale unsupervised language models (LMs) have the ability to learn broad world knowledge and some reasoning skills, but controlling their behavior precisely is a challenge due to the completely unsupervised nature of their training. Existing methods for achieving steerability involve collecting human labels of the relative quality of model generations and fine-tuning the unsupervised LM to align with these preferences, often with reinforcement learning from human feedback (RLHF). However, RLHF is complex and unstable, involving fitting a reward model that reflects human preferences and then fine-tuning the large unsupervised LM using reinforcement learning to maximize this estimated reward without drifting too far from the original model. In response, researchers have developed Direct Preference Optimization (DPO), which leverages a mapping between reward functions and optimal policies to optimize constrained reward maximization problems exactly with a single stage of policy training. Essentially solving a classification problem on human preference data, DPO eliminates the need for fitting a reward model, sampling from the LM during fine-tuning, or significant hyperparameter tuning. Experiments show that DPO can fine-tune LMs to align with human preferences as well as or better than existing methods. Notably, fine-tuning with DPO exceeds RLHF's ability to control sentiment of generations and improves response quality in summarization and single-turn dialogue while being substantially simpler to implement and train. The experiments explore three different open-ended text generation tasks: controlled sentiment generation using movie reviews from IMDb dataset; summarization using Reddit TL;DR summarization dataset; and single-turn dialogue using Anthropic Helpful and Harmless dialogue dataset. The evaluation uses two different approaches: analyzing effectiveness in optimizing constrained reward maximization objective by its frontier of achieved reward in controlled sentiment generation setting; evaluating algorithms with their win rate against baseline policy using GPT-4 as proxy for human evaluation of summary quality and response helpfulness in summarization and single-turn dialogue settings respectively. The results show that DPO tends to perform as well or better than strong baselines like RLHF with PPO and returning the best of N sampled trajectories under a learned reward function, with almost no tuning of hyperparameters. Additionally, DPO is the only method that improves over chosen summaries in the Anthropic-HH one-step dialogue task, while GPT-4 judgments correlate strongly with humans.
- Large-scale unsupervised language models (LMs) can learn broad world knowledge and reasoning skills, but controlling their behavior is challenging due to the unsupervised nature of their training.
- Existing methods for controlling LMs involve collecting human labels and fine-tuning the model with reinforcement learning from human feedback (RLHF), which is complex and unstable.
- Direct Preference Optimization (DPO) eliminates the need for fitting a reward model, sampling from the LM during fine-tuning, or significant hyperparameter tuning by solving a classification problem on human preference data.
- Experiments show that DPO can fine-tune LMs to align with human preferences as well as or better than existing methods, exceeding RLHF's ability to control sentiment of generations and improving response quality in summarization and single-turn dialogue while being substantially simpler to implement and train.
- The experiments explore three different open-ended text generation tasks: controlled sentiment generation using movie reviews from IMDb dataset; summarization using Reddit TL;DR summarization dataset; and single-turn dialogue using Anthropic Helpful and Harmless dialogue dataset.
- The evaluation uses two different approaches: analyzing effectiveness in optimizing constrained reward maximization objective by its frontier of achieved reward in controlled sentiment generation setting; evaluating algorithms with their win rate against baseline policy using GPT-4 as proxy for human evaluation of summary quality and response helpfulness in summarization and single-turn dialogue settings respectively.
- The results show that DPO tends to perform as well or better than strong baselines like RLHF with PPO and returning the best of N sampled trajectories under a learned reward function, with almost no tuning of hyperparameters. Additionally, DPO is the only method that improves over chosen summaries in the Anthropic-HH one-step dialogue task, while GPT-4 judgments correlate strongly with humans.
Large-scale unsupervised language models (LMs) are computer programs that can learn a lot of information about the world and how to reason, but it is hard to control what they say or do because they learn on their own. People have tried different ways to control these models, like giving them feedback from humans, but it can be complicated and not always work well. A new method called Direct Preference Optimization (DPO) helps control these models by using human preferences instead of rewards or feedback. Experiments show that DPO works well for making the models say things that people like in different tasks like writing summaries or having conversations.
Definitions
- Large-scale unsupervised language models: computer programs that can understand and generate language without being specifically taught
- Reinforcement learning: a type of machine learning where an algorithm learns through trial and error by receiving rewards or punishments for certain actions
- Hyperparameters: settings in a machine learning model that affect how it learns
- Fine-tuning: adjusting a pre-trained model to perform better on a specific task
Unsupervised Language Models: Controlling Behavior with Direct Preference Optimization
Large-scale unsupervised language models (LMs) have the ability to learn broad world knowledge and some reasoning skills, but controlling their behavior precisely is a challenge due to the completely unsupervised nature of their training. Existing methods for achieving steerability involve collecting human labels of the relative quality of model generations and fine-tuning the unsupervised LM to align with these preferences, often with reinforcement learning from human feedback (RLHF). However, RLHF is complex and unstable, involving fitting a reward model that reflects human preferences and then fine-tuning the large unsupervised LM using reinforcement learning to maximize this estimated reward without drifting too far from the original model. In response, researchers have developed Direct Preference Optimization (DPO), which leverages a mapping between reward functions and optimal policies to optimize constrained reward maximization problems exactly with a single stage of policy training.
What is DPO?
Essentially solving a classification problem on human preference data, DPO eliminates the need for fitting a reward model, sampling from the LM during fine-tuning, or significant hyperparameter tuning. With DPO, there is no need for an intermediate step in which rewards are estimated based on user feedback; instead it directly optimizes for user preferences by mapping them into an optimal policy space. This allows users to control what they want out of their language models more precisely than ever before while still maintaining stability in its performance.
Experiments
Experiments show that DPO can fine-tune LMs to align with human preferences as well as or better than existing methods. Notably, fine-tuning with DPO exceeds RLHF's ability to control sentiment of generations and improves response quality in summarization and single-turn dialogue while being substantially simpler to implement and train. The experiments explore three different open-ended text generation tasks: controlled sentiment generation using movie reviews from IMDb dataset; summarization using Reddit TL;DR summarization dataset; and single-turn dialogue using Anthropic Helpful and Harmless dialogue dataset.
Evaluation
The evaluation uses two different approaches: analyzing effectiveness in optimizing constrained reward maximization objective by its frontier of achieved reward in controlled sentiment generation setting; evaluating algorithms with their win rate against baseline policy using GPT-4 as proxy for human evaluation of summary quality and response helpfulness in summarization and single-turn dialogue settings respectively. The results show that DPO tends to perform as well or better than strong baselines like RLHF with PPO and returning the best of N sampled trajectories under a learned reward function, with almost no tuning of hyperparameters. Additionally, DPO is the only method that improves over chosen summaries in the Anthropic HH one step dialogue task while GPT 4 judgments correlate strongly with humans .
Conclusion
Direct Preference Optimizaton (DPO) offers an effective way for controlling behavior within large scale unsupervised language models without sacrificing accuracy or stability compared to existing methods such as Reinforcement Learning From Human Feedback (RLFH). It eliminates many steps involved in traditional methods such as fitting rewards models or sampling from LMs during finetuning while still providing excellent results across multiple tasks including controlled sentiment analysis ,summarisation ,and single turn dialogues . By leveraging mappings between rewards functions & optimal policies ,DPO provides users greater precision when controlling behaviour within language models & helps ensure stability across all tasks .
