# Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

**Authors:** Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston  
**arXiv:** [2204.05862](https://arxiv.org/abs/2204.05862)  
**Date:** 2022-04-12  
**Summary page:** https://www.summarizepaper.com/en/paper/2204.05862v1/

## Key points

- The article discusses the use of preference modeling and reinforcement learning from human feedback (RLHF) to improve language models as helpful assistants.
- Alignment training using RLHF improves performance on almost all natural language processing evaluations, including specialized skills like python coding and summarization.
- An iterated online mode of training is explored where preference models and RL policies are updated weekly with fresh human feedback data, efficiently improving datasets and models.
- The robustness of RLHF training is investigated, identifying a roughly linear relation between the RL reward and the square root of the KL divergence between the policy and its initialization.
- Peripheral analyses are performed on calibration, competing objectives, and the use of OOD detection.
- Large preference models trained on a mixture of human-human (HH) and learning-to-summarize (LtS) datasets perform equally well on both.
- RLHF decreases the performance of small code models but improves larger ones when combined with natural language alignment for coding tasks.
- Improvements in performance from RLHF are modest in all evaluations but valuable for finetuning language models for specialized skills like coding or summarization.
- Simply prompting a base code model performs slightly better than using RLHF alone.
- Sam Ge is thanked for his contributions to this research.

## Summary

The article discusses the application of preference modeling and reinforcement learning from human feedback (RLHF) to finetune language models as helpful and harmless assistants. The authors find that this alignment training improves performance on almost all natural language processing evaluations, including specialized skills such as python coding and summarization. They explore an iterated online mode of training where preference models and RL policies are updated weekly with fresh human feedback data, efficiently improving datasets and models. The authors also investigate the robustness of RLHF training and identify a roughly linear relation between the RL reward and the square root of the KL divergence between the policy and its initialization. They perform peripheral analyses on calibration, competing objectives, and the use of OOD detection, compare their models with human writers, and provide samples from their models using prompts appearing in recent related work. In one experiment, large preference models trained on a mixture of human-human (HH) and learning-to-summarize (LtS) datasets perform equally well on both. This suggests that there is no cost to mixing HH with specific skill evaluation like summarization quality. In another experiment, they test whether natural language alignment can be combined with coding without compromising performance. The authors find that RLHF decreases the performance of small code models but improves larger ones. Overall, while improvements in performance from RLHF are modest in all evaluations, it is still valuable for finetuning language models for specialized skills like coding or summarization. The authors emphasize that simply prompting a base code model performs slightly better than using RLHF alone. Finally, they thank Sam Ge for his contributions to this research.