# RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback

**Authors:** Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, Abhinav Rastogi  
**arXiv:** [2309.00267](https://arxiv.org/abs/2309.00267)  
**Date:** 2023-09-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.00267v1/

## Key points

- Reinforcement learning from human feedback (RLHF) and reinforcement learning from AI feedback (RLAIF) compared for aligning large language models (LLMs) with human preferences
- RLHF and RLAIF both result in similar improvements in summarization task
- Human evaluators prefer generations from both approaches over a baseline supervised fine-tuned model in about 70% of cases
- Humans show equal preference for RLAIF vs. RLHF summaries when rated
- RLAIF appears less likely to produce hallucinations compared to RLHF
- RLHF sometimes produces less coherent or grammatical summaries compared to RLAIF
- LLMs have shown impressive performance across various NLP tasks
- RL has been effective for optimization in related work
- RL from human feedback used successfully for aligning LLMs with human preferences in summarization, instruction following, dialogue, and question answering tasks
- LLMs used for data generation and augmentation purposes
- Experiments conducted using the filtered Reddit TL;DR dataset curated by OpenAI
- Human preference dataset curated using pairwise comparisons of candidate summaries generated by different policies
- Both RLHF and RLAIF yield high-quality summaries preferred by humans with some differences in hallucinations and coherence/grammar issues
- RLAIF can offer a scalable solution to the limitations of RLHF

## Summary

Reinforcement learning from human feedback (RLHF) and reinforcement learning from AI feedback (RLAIF) are compared in their ability to align large language models (LLMs) with human preferences. The study finds that both RLHF and RLAIF result in similar improvements in the task of summarization. Human evaluators prefer generations from both approaches over a baseline supervised fine-tuned model in about 70% of cases. When asked to rate RLAIF vs. RLHF summaries, humans show equal preference for both. However, there are some observed differences between the two approaches. RLAIF appears less likely to produce hallucinations, while RLHF sometimes produces less coherent or grammatical summaries. Overall, both approaches produce high-quality summaries that are relatively similar. In related work, LLMs have shown impressive performance across various NLP tasks and RL has been effective for optimization. RL from human feedback has been used to align LLMs with human preferences and has been successfully applied to summarization, instruction following, dialogue and question answering tasks. LLMs have also been used for data generation and augmentation purposes. Recent work introduced the idea of RL from AI feedback (RLAIF), which combines LLM-labeled preferences with human-labeled preferences to optimize for helpfulness and harmlessness objectives. The experiments conducted in this study use the filtered Reddit TL;DR dataset curated by OpenAI which contains posts from Reddit alongside summaries written by the original authors. A human preference dataset was also curated from this dataset using pairwise comparisons of candidate summaries generated by different policies. Overall, this study provides insights into the effectiveness of RLHF and RLAIF for aligning LLMs with human preferences in the task of summarization. Both approaches yield high-quality summaries preferred by humans with some differences in terms of hallucinations and coherence/grammar issues. These findings suggest that RLAIF can offer a scalable solution to the limitations of RLHF.