# Let's Verify Step by Step

**Authors:** Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, Karl Cobbe  
**arXiv:** [2305.20050](https://arxiv.org/abs/2305.20050)  
**Date:** 2023-05-31  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.20050v1/

## Key points

- Significant advancements in large language models' ability to perform complex multi-step reasoning
- Models still often make logical errors
- Two approaches to address this issue: outcome supervision and process supervision
- Outcome supervision provides feedback for the final result of a model's reasoning process
- Process supervision provides feedback for each intermediate step of the reasoning process
- Both methods are crucial due to the high cost of human feedback
- Study investigates the effectiveness of process supervision compared to outcome supervision in training models on the MATH dataset
- Process-supervised model successfully solves 78% of problems from a representative subset of the MATH test set
- Active learning techniques greatly enhance the efficacy of process supervision
- Authors release PRM800K dataset containing 800000 step-level human feedback labels used to train their best reward model
- Process supervision yields superior results compared to outcome supervision on the challenging MATH dataset
- Research contributes valuable insights to developing more reliable language models capable of accurate complex reasoning tasks

## Summary

In recent years, there have been significant advancements in large language models' ability to perform complex multi-step reasoning. However, even the most advanced models still often make logical errors. To address this issue and train more reliable models, researchers have explored two approaches: outcome supervision and process supervision. Outcome supervision involves providing feedback for the final result of a model's reasoning process. On the other hand, process supervision provides feedback for each intermediate step of the reasoning process. Both methods are crucial for training reliable models, considering the high cost of human feedback. While previous studies have begun comparing these two approaches, many questions remain unanswered. In this study, conducted by authors Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever and Karl Cobbe from OpenAI; they investigate the effectiveness of process supervision compared to outcome supervision in training models to solve problems from the challenging MATH dataset. The results of their investigation demonstrate that process supervision significantly outperforms outcome supervision. The process-supervised model successfully solves 78% of problems from a representative subset of the MATH test set. Additionally they show that active learning techniques greatly enhance the efficacy of process supervision. To support further research in this area and facilitate reproducibility of their findings; the authors release PRM800K—a comprehensive dataset containing 800000 step-level human feedback labels used to train their best reward model. This study sheds light on the importance of carefully comparing different training methods for language models' multi-step reasoning abilities. By demonstrating that process supervision yields superior results compared to outcome supervision on the challenging MATH dataset and showcasing how active learning can further improve performance; this research contributes valuable insights to developing more reliable language models capable of accurate complex reasoning tasks.