# Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning

**Authors:** Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, Aviral Kumar  
**arXiv:** [2410.08146](https://arxiv.org/abs/2410.08146)  
**Date:** 2024-10-10  
**Summary page:** https://www.summarizepaper.com/en/paper/2410.08146v1/

## Key points

- Authors explore the use of process reward models (PRMs) to enhance reasoning in large language models
- PRMs offer feedback at each step of a multi-step reasoning trace, improving credit assignment compared to outcome reward models (ORMs)
- Designing process rewards effectively involves measuring progress by evaluating the change in likelihood of producing a correct response before and after taking a step
- Progress should be evaluated under a prover policy distinct from the base policy, enhancing exploration during test-time search and online RL
- Training process advantage verifiers (PAVs) to predict progress under provers shows higher accuracy and compute efficiency compared to ORMs
- Online RL with dense rewards from PAVs achieves significant gains in sample efficiency and accuracy compared to ORMs

## Summary

In their paper titled "Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning," authors Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar explore the use of process reward models (PRMs) to enhance reasoning in large language models. PRMs offer feedback at each step of a multi-step reasoning trace and can improve credit assignment compared to outcome reward models (ORMs) that only provide feedback at the final step. However, collecting dense human labels for each step is not practical on a large scale. To address this challenge and improve the base policy through search against a PRM or using it as dense rewards for reinforcement learning (RL), the authors investigate how to design process rewards effectively. Their key insight is that the process reward for a step should measure progress by evaluating the change in likelihood of producing a correct response before and after taking the step. This aligns with the notion of step-level advantages in RL. Importantly, this progress should be evaluated under a prover policy that is distinct from the base policy. The authors theoretically characterize good provers and demonstrate that optimizing process rewards from such provers enhances exploration during test-time search and online RL. They find that even weak prover policies can significantly enhance a stronger base policy. To validate their claims, they train process advantage verifiers (PAVs) to predict progress under these provers. Comparing their approach to ORMs, test-time search against PAVs shows over 8% higher accuracy and 1.5-5 times more compute efficiency. Furthermore, online RL with dense rewards from PAVs achieves significant gains in sample efficiency (5-6 times) and accuracy (over 6%) compared to ORMs. Overall, this research highlights the importance of designing effective process rewards for improving reasoning in large language models and demonstrates significant advancements in exploration during test-time search and online RL through the use of automated process verifiers like PAVs.