# Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Training

**Authors:** Youssef Mroueh, Nicolas Dupuis, Brian Belgodere, Apoorva Nitsure, Mattia Rigotti, Kristjan Greenewald, Jiri Navratil, Jerret Ross, Jesus Rios  
**arXiv:** [2505.22257](https://arxiv.org/abs/2505.22257)  
**Date:** 2025-05-28  
**Summary page:** https://www.summarizepaper.com/en/paper/2505.22257v2/

## Key points

- Group Relative Policy Optimization (GRPO) explored in both on-policy and off-policy optimization scenarios
- Tailoring GRPO to the off-policy setting leads to improved reward outcomes
- Effectiveness of utilizing clipped surrogate objectives in the off-policy version of GRPO
- Off-policy GRPO significantly outperforms or performs comparably to its on-policy counterpart
- On-Policy and Off-Policy Actor-Critic Methods combine policy gradients with value function estimation for enhanced learning efficiency
- Off-policy variants introduce importance weighting techniques for stable updates from off-policy data
- Mixing on-policy and off-policy methods through approaches like P3O aims to leverage stability with efficiency
- Off-Policy RLHF and other variants of GRPO within the iterative DPO framework for improved convergence over epochs

## Summary

, , , , 

In this study, we explore Group Relative Policy Optimization (GRPO) in both on-policy and off-policy optimization scenarios. Our inspiration comes from recent advancements in off-policy Proximal Policy Optimization (PPO), which have shown improvements in training stability, sampling efficiency, and memory utilization. A recent analysis of GRPO has also suggested that estimating the advantage function using off-policy samples could be beneficial. Building upon these insights, we tailor GRPO to the off-policy setting and demonstrate that both on-policy and off-policy GRPO objectives lead to improved reward outcomes. Our findings highlight the effectiveness of utilizing clipped surrogate objectives in the off-policy version of GRPO. By comparing the empirical performance of reinforcement learning with verifiable rewards post-training using both variants of GRPO, our results reveal that off-policy GRPO either significantly outperforms or performs comparably to its on-policy counterpart. We also delve into the landscape of On-Policy and Off-Policy Actor-Critic Methods, which combine policy gradients with value function estimation for enhanced learning efficiency. Off-policy variants such as Off-Policy Actor-Critic algorithm and ACER introduce importance weighting techniques to enable stable updates from off-policy data. Additionally, mixing on-policy and off-policy methods through approaches like P3O aims to leverage the stability of on-policy updates with the efficiency of off-policy learning. Furthermore, we discuss Off-Policy RLHF and other variants of GRPO introduced within the iterative DPO framework for improved convergence over epochs. This comprehensive exploration sheds light on valuable insights into optimizing policies through a combination of on- and off-policy strategies for efficient reinforcement learning outcomes.