# ORPO: Monolithic Preference Optimization without Reference Model

**Authors:** Jiwoo Hong, Noah Lee, James Thorne  
**arXiv:** [2403.07691](https://arxiv.org/abs/2403.07691)  
**Date:** 2024-03-12  
**Summary page:** https://www.summarizepaper.com/en/paper/2403.07691v2/

## Key points

- Authors Jiwoo Hong, Noah Lee, and James Thorne focus on preference alignment algorithms for language models
- Emphasize the importance of supervised fine-tuning (SFT) for successful convergence in preference alignment
- Introduce ORPO (Odds Ratio Preference Optimization), a monolithic algorithm that eliminates the need for an additional preference alignment phase
- ORPO uses odds ratio to contrast favored and disfavored styles during SFT across various model sizes
- Demonstrated effectiveness of ORPO through empirical and theoretical analysis on large-scale language models like Phi-2, Llama-2, and Mistral
- ORPO surpasses state-of-the-art models with more than 7B and 13B parameters, achieving significant performance improvements in various evaluations
- Authors provide code and model checkpoints for Mistral-ORPO-alpha (7B) and Mistral-ORPO-beta (7B)
- Study highlights the potential of ORPO as a powerful tool for enhancing model performance without relying on reference models

## Summary

In their paper titled "ORPO: Monolithic Preference Optimization without Reference Model," authors Jiwoo Hong, Noah Lee, and James Thorne delve into the realm of preference alignment algorithms for language models. Recent advancements have shown promise in this area; however, the authors emphasize the continued importance of supervised fine-tuning (SFT) to ensure successful convergence. They highlight the significance of SFT within the context of preference alignment and note that even a minor penalty for the disfavored generation style can lead to effective preference-aligned SFT. Building upon this foundation, the authors introduce an innovative approach called ORPO (Odds Ratio Preference Optimization), which is a reference model-free monolithic algorithm. This novel method eliminates the need for an additional preference alignment phase and showcases how odds ratio can be a sensible choice for contrasting favored and disfavored styles during SFT across various model sizes ranging from 125M to 7B parameters. Through empirical and theoretical analysis, the authors demonstrate the effectiveness of ORPO by fine-tuning large-scale language models such as Phi-2 (2.7B), Llama-2 (7B), and Mistral (7B) on UltraFeedback data alone. Surpassing state-of-the-art models with more than 7B and 13B parameters, ORPO achieves remarkable performance improvements, including up to 12.20% on AlpacaEval_2.0, 66.19% on IFEval at instruction-level loose evaluation, and a score of 7.32 in MT-Bench evaluation. The authors also provide code and model checkpoints for Mistral-ORPO-alpha (7B) and Mistral-ORPO-beta (7B), further contributing to the research community's understanding of preference optimization in language modeling tasks. This comprehensive study sheds light on the potential of ORPO as a powerful tool for enhancing model performance without relying on reference models, paving the way for future advancements in preference-aligned fine-tuning strategies.