# Exploring Reasoning Reward Model for Agents

**Authors:** Kaixuan Fan, Kaituo Feng, Manyuan Zhang, Tianshuo Peng, Zhixun Li, Yilei Jiang, Shuang Chen, Peng Pei, Xunliang Cai, Xiangyu Yue  
**arXiv:** [2601.22154](https://arxiv.org/abs/2601.22154)  
**Date:** 2026-01-29  
**Summary page:** https://www.summarizepaper.com/en/paper/2601.22154v1/

## Key points

- Significant strides in Agentic Reinforcement Learning (Agentic RL) have been made, enabling agents to engage in complex reasoning and utilize tools.
- Existing methods heavily rely on sparse outcome-based rewards for training, leading to suboptimal training outcomes by hindering the quality of intermediate reasoning processes.
- The Agent Reasoning Reward Model (Agent-RRM) is introduced as a sophisticated reward framework that provides structured feedback for agentic trajectories, consisting of an explicit reasoning trace, focused critique for refinement guidance, and an overall score for comprehensive process performance evaluation.
- Three integration strategies are explored: Reagent-C (text-augmented refinement), Reagent-R (reward-augmented guidance), and Reagent-U (unified feedback integration), with Reagent-U showing substantial performance improvements across 12 diverse benchmarks.
- The Reagent-U strategy achieves notable success rates of 43.7% on GAIA and 46.2% on WebWalkerQA, highlighting the effectiveness of the reasoning reward model and associated training schemes.
- The authors have released code implementations, models, and datasets to facilitate further advancements in agent reasoning capabilities and reinforcement learning methodologies.

## Summary

Significant strides have been made in the realm of Agentic Reinforcement Learning (Agentic RL), empowering agents to engage in complex reasoning and utilize tools. However, a prevalent issue persists where most existing methods heavily rely on sparse outcome-based rewards for training. This limitation hampers the ability to discern the quality of intermediate reasoning processes, ultimately resulting in suboptimal training outcomes. To address this challenge, this paper introduces the Agent Reasoning Reward Model (Agent-RRM), a sophisticated reward framework that generates structured feedback for agentic trajectories. The model encompasses three key components: an explicit reasoning trace, a focused critique that offers refinement guidance by pinpointing reasoning flaws, and an overall score that evaluates process performance comprehensively. To explore the efficacy of the Agent-RRM, three integration strategies are systematically investigated: Reagent-C (text-augmented refinement), Reagent-R (reward-augmented guidance), and Reagent-U (unified feedback integration). Through extensive evaluations across 12 diverse benchmarks, it is demonstrated that the Reagent-U strategy yields substantial performance improvements. Notably, this approach achieves impressive results with a 43.7% success rate on GAIA and a remarkable 46.2% on WebWalkerQA, underscoring the effectiveness of the reasoning reward model and associated training schemes. Moreover, to foster further advancements in this field, the authors have generously released code implementations, models, and datasets associated with their research endeavors. The comprehensive nature of this work not only sheds light on enhancing agent reasoning capabilities but also provides valuable resources for future exploration and innovation in reinforcement learning methodologies.