# rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking

**Authors:** Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, Mao Yang  
**arXiv:** [2501.04519](https://arxiv.org/abs/2501.04519)  
**Date:** 2025-01-08  
**Summary page:** https://www.summarizepaper.com/en/paper/2501.04519v1/

## Key points

- Introduction of rStar-Math, a novel approach utilizing small language models (SLMs) for math reasoning capabilities comparable to or surpassing OpenAI o1
- Utilization of "deep thinking" via Monte Carlo Tree Search (MCTS) for test-time search guided by an SLM-based process reward model
- Novel code-augmented CoT data synthesis method using MCTS rollouts for verified reasoning trajectories
- Unique process reward model training method eliminating naive step-level score annotations
- Self-evolution strategy enhancing reasoning capabilities through iterative evolution of policy SLM and PPM
- Significant improvement in SLMs' math reasoning abilities through four rounds of self-evolution with millions of synthesized solutions for math problems
- Performance enhancement on benchmark tests such as MATH benchmark and USA Math Olympiad (AIME)
- Methodology involving two 7B SLMs generating higher-quality training data through extensive MCTS rollouts on accessible hardware
- Implementation of innovative strategies like code-augmented CoT synthetic methods and self-evolution recipes to overcome challenges faced by SLMs compared to advanced models like GPT-4
- Detailed explanation provided on generating step-by-step verified reasoning trajectories with per-step Q-value annotations using MCTS and code-augmented CoT synthesis methods
- Enhancement of Q-value accuracy reliability through filtering out low-quality generations and conducting extensive rollouts for constructing high-quality training sets
- Showcase of significant advancements in enhancing small language models' math reasoning capabilities through innovative methodologies and iterative self-evolution strategies, achieving impressive results in mathematics problem-solving.

## Summary

In the study, the researchers introduce rStar-Math, a novel approach that demonstrates how small language models (SLMs) can achieve math reasoning capabilities comparable to or even surpassing those of OpenAI o1 without the need for distillation from superior models. This is achieved through the utilization of "deep thinking" via Monte Carlo Tree Search (MCTS), where a math policy SLM conducts test-time search guided by an SLM-based process reward model. The researchers introduce three key innovations to address challenges in training the two SLMs: 

Firstly, they propose a novel code-augmented CoT data synthesis method that utilizes extensive MCTS rollouts to generate step-by-step verified reasoning trajectories for training the policy SLM. Secondly, they present a unique process reward model training method that eliminates the need for naive step-level score annotations, resulting in a more effective process preference model (PPM). Lastly, they implement a self-evolution strategy where both the policy SLM and PPM are built from scratch and iteratively evolved to enhance their reasoning capabilities. Through four rounds of self-evolution with millions of synthesized solutions for 747k math problems, rStar-Math significantly boosts SLMs' math reasoning abilities to state-of-the-art levels. On benchmark tests such as MATH benchmark, it improves Qwen2.5-Math-7B from 58.8% to 90.0% and Phi3-mini-3.8B from 41.4% to 86.4%, outperforming o1-preview by +4.5% and +0.9%. Additionally, on the USA Math Olympiad (AIME), rStar-Math successfully solves an average of 53.3% (8/15) of problems, placing it among the top 20% of high school math students. The methodology employed involves using two 7B SLMs (policy SLM and PRM) to generate higher-quality training data through extensive MCTS rollouts on accessible hardware. Despite challenges faced by SLMs in generating correct solutions and intermediate steps compared to advanced models like GPT-4, innovative strategies such as code-augmented CoT synthetic methods and self-evolution recipes are implemented to improve performance on challenging problems. Furthermore, a detailed explanation is provided on how step-by-step verified reasoning trajectories with per-step Q-value annotations are generated using MCTS and code-augmented CoT synthesis methods. By filtering out low-quality generations and conducting extensive rollouts, the reliability of Q-value accuracy is enhanced in order to construct a high-quality training set for improving SLM performance on complex mathematical problems. Overall, rStar-Math showcases significant advancements in enhancing small language models' math reasoning capabilities through innovative methodologies and iterative self-evolution strategies, ultimately achieving impressive results on various benchmark tests and competitions within the field of mathematics problem-solving.