Results of the summarizing process for the arXiv paper: 2512.01374v1

In their paper titled "Stabilizing Reinforcement Learning with LLMs: Formulation and Practices," authors Chujie Zheng, Kai Dang, Bowen Yu, Mingze Li, Huiqiang Jiang, Junrong Lin, Yuqiong Liu, An Yang, Jingren Zhou, and Junyang Lin propose a novel formulation for reinforcement learning (RL) using large language models. They explore how the true sequence-level reward can be optimized through a surrogate token-level objective in policy gradient methods like REINFORCE. By conducting a first-order approximation analysis, the authors demonstrate that this surrogate objective becomes increasingly valid when minimizing both training-inference discrepancy and policy staleness. The study sheds light on the importance of various techniques in stabilizing RL training including importance sampling correction, clipping and Routing Replay for Mixture-of-Experts (MoE) models. Through extensive experiments involving a 30B MoE model and hundreds of thousands of GPU hours,the authors find that on-policy training with the basic policy gradient algorithm and importance sampling correction yields the highest training stability. They also highlight that introducing off-policy updates to accelerate convergence requires combining clipping and Routing Replay to mitigate instability stemming from policy staleness. Once training is stabilized using these techniques,prolonged optimization consistently leads to comparable final performance regardless of cold-start initialization. The authors hope that by sharing their insights and developed strategies for stable RL training in this paper will facilitate future research in this area.

• Authors propose a novel formulation for reinforcement learning (RL) using large language models
• Surrogate token-level objective in policy gradient methods like REINFORCE optimizes true sequence-level reward
• Importance of techniques such as importance sampling correction, clipping, and Routing Replay for stabilizing RL training highlighted
• On-policy training with basic policy gradient algorithm and importance sampling correction yields highest training stability
• Off-policy updates to accelerate convergence require combining clipping and Routing Replay to mitigate instability from policy staleness
• Prolonged optimization leads to comparable final performance regardless of cold-start initialization

Reinforcement learning (RL) has been a popular area of research in recent years due to its potential for solving complex decision-making problems. However, one major challenge in RL is the instability of training, which can lead to poor performance and slow convergence. In their paper titled "Stabilizing Reinforcement Learning with LLMs: Formulation and Practices," Chujie Zheng et al. propose a novel formulation for reinforcement learning using large language models (LLMs) to address this issue.

The authors start by highlighting the importance of optimizing the true sequence-level reward in RL tasks. They explain that traditional policy gradient methods like REINFORCE use a token-level objective as a surrogate for the sequence-level reward, which can result in suboptimal performance. To overcome this limitation, they introduce a new formulation that leverages LLMs to optimize the true sequence-level reward directly.

To support their proposed formulation, the authors conduct a first-order approximation analysis and show that it becomes increasingly valid when minimizing both training-inference discrepancy and policy staleness. This finding highlights the importance of these two factors in stabilizing RL training.

Next, they delve into various techniques that can be used to stabilize RL training, including importance sampling correction, clipping, and Routing Replay for Mixture-of-Experts (MoE) models. Through extensive experiments involving a 30B MoE model and hundreds of thousands of GPU hours,the authors find that on-policy training with basic policy gradient algorithm and importance sampling correction yields the highest stability during training.

They also demonstrate that introducing off-policy updates can accelerate convergence but requires combining clipping and Routing Replay to mitigate instability caused by policy staleness. This finding suggests that careful consideration must be given when incorporating off-policy updates into RL algorithms.

Furthermore, the authors highlight an interesting observation from their experiments - once training is stabilized using these techniques, prolonged optimization consistently leads to comparable final performance regardless of cold-start initialization. This finding is significant as it suggests that the proposed formulation and techniques can be applied to a wide range of RL tasks without the need for task-specific tuning.

Overall, this paper provides valuable insights into stabilizing RL training using LLMs. By sharing their findings and developed strategies, the authors hope to facilitate future research in this area. They also highlight potential directions for further exploration, such as incorporating more advanced optimization methods and exploring different types of LLMs.

In conclusion, "Stabilizing Reinforcement Learning with LLMs: Formulation and Practices" is a well-written and informative paper that sheds light on the importance of optimizing true sequence-level reward in RL tasks. The proposed formulation and techniques have been extensively tested through experiments, providing strong evidence for their effectiveness in stabilizing RL training. This paper will undoubtedly serve as a valuable resource for researchers working in this field and pave the way for future advancements in reinforcement learning with large language models.