# Self-Rewarding Language Models

**Authors:** Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, Jason Weston  
**arXiv:** [2401.10020](https://arxiv.org/abs/2401.10020)  
**Date:** 2024-01-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2401.10020v1/

## Key points

- Achieving superhuman agents through the use of superhuman feedback for training
- Training reward models based on human preferences is limited by human performance levels and frozen reward models
- Proposal of Self-Rewarding Language Models where the language model provides its own rewards through LLM-as-a-Judge prompting during training
- Improvement in instruction following ability and generation of high-quality rewards during Iterative DPO training
- Impressive results achieved on Llama 2 70B, outperforming existing systems on the AlpacaEval 2.0 leaderboard
- Possibilities for models to continuously enhance instruction following ability and reward generation capability through self-rewarding mechanisms
- Potential for language models to surpass human performance levels and improve their own capabilities using these mechanisms

## Summary

The paper titled "Self-Rewarding Language Models" explores the concept of achieving superhuman agents through the use of superhuman feedback for training. The current approach involves training reward models based on human preferences, but this method is limited by human performance levels and frozen reward models that cannot improve during language model (LLM) training. To overcome these limitations, the authors propose Self-Rewarding Language Models where the language model itself provides its own rewards through LLM-as-a-Judge prompting during training. The study demonstrates that during Iterative DPO (Deep Policy Optimization) training, not only does the instruction following ability of the language model improve, but it also becomes capable of generating high-quality rewards for itself. Through three iterations of their approach on Llama 2 70B, impressive results are achieved. The resulting model outperforms several existing systems on the AlpacaEval 2.0 leaderboard including Claude 2, Gemini Pro, and GPT-4 0613. While considered a preliminary study, this work opens up possibilities for models that can continuously enhance both their instruction following ability and reward generation capability through self-rewarding mechanisms. In the future, language models may surpass human performance levels and continually improve their own capabilities by leveraging these mechanisms.