# Offline Reinforcement Learning for LLM Multi-Step Reasoning

**Authors:** Huaijie Wang, Shibo Hao, Hanze Dong, Shenao Zhang, Yilin Bao, Ziran Yang, Yi Wu  
**arXiv:** [2412.16145](https://arxiv.org/abs/2412.16145)  
**Date:** 2024-12-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2412.16145v1/

## Key points

- Authors address the challenge of enhancing multi-step reasoning ability of large language models (LLMs) through offline reinforcement learning (RL)
- Direct Preference Optimization (DPO) is not well-suited for multi-step reasoning tasks due to reliance on paired preference data and uniform treatment of all tokens
- Proposed solution: OREO (Offline Reasoning Optimization), an offline RL method leveraging insights from maximum entropy reinforcement learning
- OREO learns a policy model and value function simultaneously by optimizing the soft Bellman Equation, reducing need for pairwise data and enabling effective credit assignment in multi-step reasoning tasks with sparse rewards
- Empirical results show OREO outperforms existing offline learning methods on various multi-step reasoning benchmarks including mathematical reasoning tasks like GSM8K and MATH, as well as embodied agent control tasks like ALFWorld
- OREO can be extended to a multi-iteration framework when additional resources are available
- Learned value function from OREO can guide tree search at no additional cost, improving performance during test time

## Summary

In their paper titled "Offline Reinforcement Learning for LLM Multi-Step Reasoning," authors Huaijie Wang, Shibo Hao, Hanze Dong, Shenao Zhang, Yilin Bao, Ziran Yang, and Yi Wu address the challenge of enhancing the multi-step reasoning ability of large language models (LLMs) through offline reinforcement learning (RL). They highlight the importance of quickly adapting LLMs to complex tasks and note that while Direct Preference Optimization (DPO) has shown promise in aligning LLMs with human preferences, it is not well-suited for multi-step reasoning tasks due to its reliance on paired preference data and uniform treatment of all tokens. To overcome these limitations, the authors propose OREO (Offline Reasoning Optimization), an offline RL method that leverages insights from maximum entropy reinforcement learning. OREO simultaneously learns a policy model and value function by optimizing the soft Bellman Equation. This approach reduces the need for collecting pairwise data and enables more effective credit assignment in multi-step reasoning tasks with sparse rewards. Empirical results demonstrate that OREO outperforms existing offline learning methods on various multi-step reasoning benchmarks, including mathematical reasoning tasks such as GSM8K and MATH, as well as embodied agent control tasks like ALFWorld. The authors also suggest that OREO can be extended to a multi-iteration framework when additional resources are available. Furthermore, the learned value function from OREO can be utilized to guide tree search at no additional cost, leading to improved performance during test time. Overall, this work presents a promising approach for enhancing LLM multi-step reasoning capabilities through offline reinforcement learning techniques.