# Reasoning Language Models: A Blueprint

**Authors:** Maciej Besta, Julia Barth, Eric Schreiber, Ales Kubicek, Afonso Catarino, Robert Gerstenberger, Piotr Nyczyk, Patrick Iff, Yueling Li, Sam Houliston, Tomasz Sternal, Marcin Copik, Grzegorz Kwaśniewski, Jürgen Müller, Łukasz Flis, Hannes Eberhard, Hubert Niewiadomski, Torsten Hoefler  
**arXiv:** [2501.11223](https://arxiv.org/abs/2501.11223)  
**Date:** 2025-01-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2501.11223v1/

## Key points

- Reasoning Language Models (RLMs) are a groundbreaking advancement in AI problem-solving capabilities
- RLMs, also known as Large Reasoning Models (LRMs), integrate advanced reasoning mechanisms into large language models (LLMs)
- Challenges faced by RLMs include high costs, proprietary constraints, and complex architectures combining RL, search heuristics, and LLMs
- Researchers led by Maciej Besta and Julia Barth propose a modular framework to organize RLM components for enhanced accessibility and scalability
- The blueprint includes diverse reasoning structures like chains, trees, graphs, and nested forms; reasoning strategies such as Monte Carlo Tree Search and Beam Search; RL concepts like policy and value models; supervision schemes like Output-Based and Process-Based Supervision
- Detailed mathematical formulations and algorithmic specifications simplify the implementation of RLMs within the framework
- Existing schemes like LLaMA-Berry, QwQ, Journey Learning, and Graph of Thoughts can be accommodated within the framework as special cases
- Practical applications of the blueprint are demonstrated through x1—a modular implementation for rapid prototyping and experimentation with RLMs
- Recommendations include multi-phase training strategies for policy and value models within RLMs while emphasizing familiar training distributions
- RLMs can seamlessly integrate into a broader LLM ecosystem encompassing tools and databases
- Efforts aim to democratize advanced reasoning capabilities across AI research communities by lowering barriers to RLM development through innovative frameworks like x1

## Summary

In the realm of artificial intelligence, Reasoning Language Models (RLMs) have emerged as a groundbreaking advancement in problem-solving capabilities. These models, also referred to as Large Reasoning Models (LRMs), such as OpenAI's o1 and o3, DeepSeek-V3, and Alibaba's QwQ, have revolutionized AI by integrating advanced reasoning mechanisms into large language models (LLMs). Despite their transformative potential, RLMs face challenges related to high costs, proprietary constraints, and complex architectures that combine Reinforcement Learning (RL), search heuristics, and LLMs in unique ways. To tackle these obstacles and enhance accessibility and scalability, a group of researchers led by Maciej Besta and Julia Barth has proposed a comprehensive blueprint for organizing RLM components within a modular framework. Drawing insights from an extensive survey and analysis of existing RLM works, this blueprint encompasses diverse reasoning structures such as chains, trees, graphs, and nested forms. It also incorporates various reasoning strategies like Monte Carlo Tree Search and Beam Search; RL concepts including policy and value models; as well as supervision schemes like Output-Based and Process-Based Supervision. Furthermore,the blueprint provides detailed mathematical formulations and algorithmic specifications aimed at simplifying the implementation of RLMs. By showcasing how existing schemes like LLaMA-Berry,QwQ,Journey Learning,and Graph of Thoughts can be accommodated within this framework as special cases,the researchers demonstrate its versatility and unifying potential. To illustrate practical applications of this blueprint in action,the researchers introduce x1—a modular implementation designed for rapid prototypingand experimentation with RLMs.Through their work on refining the blueprint alongside a thorough literature review process led by Eric Schreiberand Ales Kubicek among others in the team—key insights have been uncovered.These include recommendations for multi-phase training strategies for policy and value models within RLMs while emphasizing the significance of utilizing familiar training distributions. Additionally,outlined is how RLMs can seamlessly integrate into a broader LLM ecosystem encompassing tools and databases. Ultimately,aiming to demystify the construction of RLMs while democratizing advanced reasoning capabilities across AI research communities—the collaborative efforts spearheaded by this team seek to bridge the gap between "rich AI" systems developed by tech giants versus "poor AI" solutions accessible to smaller entities.By lowering barriers to RLM development through innovative frameworks like x1—this initiative sets out to foster creativity in AI research while promoting inclusivity within the field.