# Direct Preference Optimization: Your Language Model is Secretly a Reward Model

**Authors:** Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn  
**arXiv:** [2305.18290](https://arxiv.org/abs/2305.18290)  
**Date:** 2023-05-29  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.18290v2/

## Key points

- The authors propose a new approach to achieving precise control over large-scale unsupervised language models (LMs)
- Existing methods for gaining steerability involve collecting human labels and using reinforcement learning from human feedback (RLHF)
- The authors introduce DPO as a new parameterization of the reward model in RLHF
- DPO allows for extraction of the optimal policy in closed form, eliminating the need for sampling from the LM during fine-tuning or significant hyperparameter tuning
- The resulting algorithm is stable, performant, and computationally lightweight
- DPO can effectively fine-tune LMs to align with human preferences as well as or better than existing methods
- It outperforms PPO-based RLHF in controlling sentiment of generations and matches or improves response quality in summarization and single-turn dialogue tasks
- DPO is simpler to implement and train compared to other preference learning algorithms like PPO
- Further evaluation is done on different text generation tasks including controlled sentiment generation, summarization, and dialogue
- DPO performs efficiently in trading off maximizing reward and minimizing KL-divergence compared to common preference learning algorithms like PPO
- It shows strong performance on larger models and more difficult RLHF tasks without requiring extensive hyperparameter tuning

## Summary

In this paper, the authors propose a new approach to achieving precise control over large-scale unsupervised language models (LMs) by introducing . While these LMs have the ability to learn broad world knowledge and some reasoning skills, their completely unsupervised training makes it difficult to control their behavior. Existing methods for gaining steerability involve collecting human labels and using reinforcement learning from human feedback (RLHF), which can be complex and unstable. To overcome these limitations, the authors introduce DPO as a new parameterization of the reward model in RLHF. This allows for extraction of the optimal policy in closed form, eliminating the need for sampling from the LM during fine-tuning or significant hyperparameter tuning. The resulting algorithm is stable, performant, and computationally lightweight. The experiments conducted show that DPO can effectively fine-tune LMs to align with human preferences as well as or better than existing methods. It outperforms PPO-based RLHF in controlling sentiment of generations and matches or improves response quality in summarization and single-turn dialogue tasks. Additionally, DPO is simpler to implement and train compared to other preference learning algorithms like PPO. Further evaluation is done on different text generation tasks including controlled sentiment generation, summarization, and dialogue. The results demonstrate that DPO performs efficiently in trading off maximizing reward and minimizing KL-divergence compared to common preference learning algorithms like PPO. It also shows strong performance on larger models and more difficult RLHF tasks without requiring extensive hyperparameter tuning. Overall, for achieving precise control over language models by directly optimizing policies based on human preferences.