# Fine-Tuning Language Models with Advantage-Induced Policy Alignment

**Authors:** Banghua Zhu, Hiteshi Sharma, Felipe Vieira Frujeri, Shi Dong, Chenguang Zhu, Michael I. Jordan, Jiantao Jiao  
**arXiv:** [2306.02231](https://arxiv.org/abs/2306.02231)  
**Date:** 2023-06-04  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.02231v1/

## Key points

- Reinforcement Learning from Human Feedback (RLHF) aligns large language models (LLMs) with human preferences
- Proximal Policy Optimization (PPO) is a widely used RLHF technique but has limitations
- Authors propose a novel algorithm called Advantage-Induced Policy Alignment (APA)
- APA utilizes squared error loss function based on estimated advantages to improve performance
- APA consistently outperforms PPO in language tasks with a separate reward model as evaluator
- APA provides more stable control over deviation from initial policy, avoiding deterministic output collapse
- Theoretical justification provided for design of APA's loss function
- APA offers effective solution for improving RLHF in language models
- Contributes to advancing reinforcement learning techniques and enhancing alignment between LLMs and human preferences

## Summary

Reinforcement Learning from Human Feedback (RLHF) has become a popular method for aligning large language models (LLMs) with human preferences. Proximal Policy Optimization (PPO) is one of the most widely used RLHF techniques but has limitations such as mode collapse, instability and poor sample efficiency. To address these issues, the authors propose a novel algorithm called Advantage-Induced Policy Alignment (APA). APA utilizes a squared error loss function based on estimated advantages to improve performance. In empirical evaluations, APA consistently outperforms PPO in language tasks by a significant margin when a separate reward model is used as the evaluator. Compared to PPO, APA provides more stable control over the deviation from the model's initial policy; this ensures that the model improves its performance without collapsing into deterministic output. The authors also provide theoretical justification for the design of their loss function. The results demonstrate that APA offers an effective solution for improving RLHF in language models and contributes to advancing reinforcement learning techniques and enhancing alignment between LLMs and human preferences.