# KTO: Model Alignment as Prospect Theoretic Optimization

**Authors:** Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, Douwe Kiela  
**arXiv:** [2402.01306](https://arxiv.org/abs/2402.01306)  
**Date:** 2024-02-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.01306v4/

## Key points

- Authors: Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, Douwe Kiela
- Concept: Model alignment in the context of prospect theory
- Highlight: Aligning Language Models (LLMs) with human feedback to incorporate biases from prospect theory like loss aversion
- Idea introduced: "human-aware losses" (HALOs) as loss functions capturing biases for successful alignment objectives like DPO
- New HALO proposed: Kahneman-Tversky Optimization (KTO) based on Kahneman & Tversky's prospect theory
- KTO focus: Maximizing utility of generations rather than log-likelihood of preferences
- Performance surprise: KTO outperforms preference-based methods despite learning only from binary signals indicating desirability
- Conclusion emphasis: Importance of inductive biases in alignment objectives and no one-size-fits-all HALO choice
- Significance: Incorporating human biases into machine learning models for more effective model alignment strategies

## Summary

In their paper titled "KTO: Model Alignment as Prospect Theoretic Optimization," Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela explore the concept of model alignment in the context of prospect theory. They highlight how aligning Language Models (LLMs) with human feedback incorporates biases from prospect theory, such as loss aversion. The authors introduce the idea of "human-aware losses" (HALOs), which are a family of loss functions that capture these biases and contribute to the success of alignment objectives like DPO over traditional methods like cross-entropy minimization. Building on Kahneman & Tversky's prospect theory, the researchers propose a new HALO called Kahneman-Tversky Optimization (KTO). Unlike existing methods that maximize log-likelihood of preferences, KTO directly focuses on maximizing the utility of generations. Surprisingly, despite learning only from binary signals indicating desirability, KTO outperforms preference-based methods across various scales from 1B to 30B. The study concludes by emphasizing the importance of inductive biases in alignment objectives and suggests that there is no one-size-fits-all HALO. The choice of the best loss function depends on the specific context and inductive biases most suitable for a given setting. This work sheds light on the significance of incorporating human biases into machine learning models for more effective model alignment strategies.