# Fine-Tuning Language Models from Human Preferences

**Authors:** Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, Geoffrey Irving  
**arXiv:** [1909.08593](https://arxiv.org/abs/1909.08593)  
**Date:** 2019-09-18  
**Summary page:** https://www.summarizepaper.com/en/paper/1909.08593v1/

## Key points

- This paper explores the application of reward learning to natural language tasks using generative pretraining of language models.
- Reward learning allows reinforcement learning (RL) to be applied to tasks where the reward is defined by human judgment.
- Incorporating reward learning for language is crucial for making RL practical and safe for real-world tasks.
- The authors focus on four natural language tasks: continuing text with positive sentiment or physically descriptive language, and summarization tasks on the TL;DR and CNN/Daily Mail datasets.
- Promising results are achieved in stylistic continuation with only 5,000 comparisons evaluated by humans.
- Models trained with 60,000 comparisons tend to copy whole sentences from the input while skipping irrelevant preamble in summarization tasks.
- This approach leads to reasonable ROUGE scores and high performance according to human labelers, but there is a concern about models exploiting simple heuristics used by labelers.
- The research demonstrates how reward learning can be effectively applied to natural language tasks using generative pretraining of language models.
- The findings highlight both the potential and limitations of using reward learning in real-world applications and provide insights into improving RL systems for language-related tasks.

## Summary

This paper explores the application of reward learning to natural language tasks using generative pretraining of language models. Reward learning allows reinforcement learning (RL) to be applied to tasks where the reward is defined by human judgment. While most previous work on reward learning has focused on simulated environments, this paper argues that incorporating reward learning for language is crucial for making RL practical and safe for real-world tasks. The authors specifically focus on four natural language tasks: continuing text with positive sentiment or physically descriptive language, and summarization tasks on the TL;DR and CNN/Daily Mail datasets. For stylistic continuation, they achieve promising results with only 5,000 comparisons evaluated by humans. When it comes to summarization, the models trained with 60,000 comparisons tend to copy whole sentences from the input while skipping irrelevant preamble. This approach leads to reasonable ROUGE scores and high performance according to human labelers. However, there is a concern that these models may be exploiting the fact that labelers rely on simple heuristics. Overall, this research demonstrates how reward learning can be effectively applied to natural language tasks using generative pretraining of language models. The findings highlight both the potential and limitations of using reward learning in real-world applications and provide insights into improving RL systems for language-related tasks.