# Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training

**Authors:** Mingjie Liu, Shizhe Diao, Jian Hu, Ximing Lu, Xin Dong, Hao Zhang, Alexander Bukharin, Shaokun Zhang, Jiaqi Zeng, Makesh Narsimhan Sreedhar, Gerald Shen, David Mosallanezhad, Di Zhang, Jonas Yang, June Yang, Oleksii Kuchaiev, Guilin Liu, Zhiding Yu, Pavlo Molchanov, Yejin Choi  
**arXiv:** [2507.12507](https://arxiv.org/abs/2507.12507)  
**Date:** 2025-07-16  
**Summary page:** https://www.summarizepaper.com/en/paper/2507.12507v1/

## Key points

- Study titled "Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training"
- Importance of utilizing verifiable reward tasks and practical techniques for training stability and generalization
- Notable improvements over strong baselines in math tasks (+14.7%), coding challenges (+13.9%), and logic puzzle performance (+54.8%)
- Critical components identified: controlled KL regularization, clipping ratio adjustments, periodic reference policy resets
- Focus on enhancing reasoning capabilities in language models through prolonged reinforcement learning
- Emphasis on scaling test-time computation through chain-of-thought reasoning and iterative exploration for significant task improvements
- Model publicly available for further research and development to unlock diverse reasoning capabilities in language models
- Potential demonstrated by the study to improve reasoning capabilities across diverse domains using advanced techniques

## Summary

In their recent study titled "Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training," Mingjie Liu, Shizhe Diao, Jian Hu, and a team of researchers explore the impact of prolonged reinforcement learning on a small language model across various reasoning domains. The study highlights the importance of utilizing verifiable reward tasks and implementing practical techniques to enhance training stability and generalization. Through their research efforts, the team achieves notable improvements over strong baselines in math tasks (+14.7%), coding challenges (+13.9%), and logic puzzle performance (+54.8%). Key components such as controlled KL regularization, clipping ratio adjustments, and periodic reference policy resets are identified as critical for unlocking long-term performance gains. By leveraging advanced techniques and strategies, the researchers pave the way for future advancements in artificial intelligence research focused on improving task performance through extended training methodologies. Reinforcement Learning, Language Models, Reasoning, Prolonged Training, Performance Enhancement

The comprehensive findings presented in this study shed light on the potential of prolonged reinforcement learning to enhance reasoning capabilities in language models across diverse domains. Building on recent advancements in reasoning-focused language models like OpenAI's O1 and DeepSeek-R1, the researchers demonstrate that scaling test-time computation through chain-of-thought reasoning and iterative exploration can lead to significant improvements in complex tasks such as mathematics and code generation. To support further exploration in this area,<Organization>, make their model publicly available for continued research and development. This allows for continued progress towards unlocking diverse reasoning capabilities in language models through prolonged reinforcement learning. Through their thorough investigation,<Organization>'s study emphasizes the importance of utilizing verifiable reward tasks,<Organization>, enhancing Group Relative Policy Optimization (GRPO), and implementing practical techniques to enhance training stability and generalization. These key components, including controlled KL regularization, clipping ratio adjustments, and periodic reference policy resets, are critical for unlocking long-term performance gains in language models. In conclusion,<Organization>'s research efforts have demonstrated the potential of prolonged reinforcement learning to significantly improve reasoning capabilities in language models across diverse domains. By leveraging advanced techniques and strategies, the team has paved the way for future advancements in artificial intelligence research focused on enhancing task performance through extended training methodologies.