# Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models

**Authors:** Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, Chenyang Shao, Yuwei Yan, Qinglong Yang, Yiwen Song, Sijian Ren, Xinyuan Hu, Yu Li, Jie Feng, Chen Gao, Yong Li  
**arXiv:** [2501.09686](https://arxiv.org/abs/2501.09686)  
**Date:** 2025-01-16  
**Summary page:** https://www.summarizepaper.com/en/paper/2501.09686v3/

## Key points

- Language is a fundamental tool for human reasoning
- Large Language Models (LLMs) are being used for complex reasoning tasks
- Introduction of "thought" concept allows LLMs to emulate human reasoning processes
- Reinforcement learning (RL) is applied to train LLMs for better reasoning capabilities
- Encouraging LLMs to engage in more extensive "thinking" during test-time improves reasoning accuracy
- Train-time and test-time scaling lead to the development of Large Reasoning Models
- OpenAI's o1 series marks a significant milestone in this research direction
- Automated data construction, learning-to-reason techniques, and test-time scaling are key components driving large reasoning model development
- Test-time enhancing techniques can further enhance LLMs' reasoning capacities by enabling strategic reasoning across solution spaces, leveraging past experiences, and optimizing workflows dynamically
- Designing robust evaluation benchmarks is crucial for documenting improvements in LLM capabilities and guiding future research directions

## Summary

Language has long been recognized as a fundamental tool for human reasoning. The emergence of Large Language Models (LLMs) has sparked considerable interest in utilizing these models to tackle complex reasoning tasks. Researchers have advanced beyond simple autoregressive token generation by introducing the concept of "thought" - a sequence of tokens representing intermediate steps in the reasoning process. This innovative approach allows LLMs to emulate intricate human reasoning processes like tree search and reflective thinking. A recent trend in learning to reason involves applying reinforcement learning (RL) to train LLMs to excel in reasoning processes. This method enables the automatic generation of high-quality reasoning trajectories through trial-and-error search algorithms, significantly enhancing LLMs' reasoning capacity by providing ample training data. Moreover, studies have shown that encouraging LLMs to engage in more extensive "thinking" with additional tokens during test-time inference can substantially improve reasoning accuracy. The combination of train-time and test-time scaling presents a new research frontier towards developing Large Reasoning Models. The introduction of OpenAI's o1 series signifies a significant milestone in this research direction. In this survey, we delve into recent advancements in LLM reasoning, starting with an introduction to the foundational background of LLMs. We then explore key technical components driving the development of large reasoning models, such as automated data construction, learning-to-reason techniques, and test-time scaling. Furthermore, we discuss how test-time enhancing techniques have the potential to further enhance LLMs' reasoning capacities by enabling them to strategically reason across solution spaces, leverage past experiences, and dynamically optimize workflows. Additionally, designing robust evaluation benchmarks is crucial for documenting improvements in LLM capabilities and guiding future research directions. We review popular benchmarks for LLM reasoning and categorize them systematically based on their taxonomy. Overall, the field is moving towards developing Large Reasoning Models that can mimic complex human-like reasoning processes effectively. By leveraging advancements in automated data construction, learning-to-reason techniques, and test-time scaling strategies, researchers are paving the way for more sophisticated and capable language models that can excel at complex reasoning tasks.