# LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoning

**Authors:** Di Zhang, Jianbo Wu, Jingdi Lei, Tong Che, Jiatong Li, Tong Xie, Xiaoshui Huang, Shufei Zhang, Marco Pavone, Yuqiang Li, Wanli Ouyang, Dongzhan Zhou  
**arXiv:** [2410.02884](https://arxiv.org/abs/2410.02884)  
**Date:** 2024-10-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2410.02884v2/

## Key points

- The LLaMA-Berry framework enhances the reasoning ability of Large Language Models (LLMs) through an advanced mathematical problem-solving approach.
- It combines Monte Carlo Tree Search (MCTS) with iterative Self-Refine to optimize the reasoning path and utilize a pairwise reward model for global evaluation of different paths.
- The Self-Refine applied to MCTS (SR-MCTS) technique promotes more efficient exploration of solution spaces compared to conventional search algorithms.
- The Pairwise Preference Reward Model (PPRM) uses an Enhanced Borda Count (EBC) method to synthesize preferences between solutions into a global ranking score for improved answers.
- Challenges in practical applications include high computational costs of methods like MCTS and Self-Refine, prompting the need for a learning-based summarizer for enhanced efficiency.
- Evaluation has focused on mathematical reasoning benchmarks, necessitating validation in broader domains such as general knowledge and symbolic logic tasks.
- Future work aims to enhance applicability by evaluating LLaMA-Berry on diverse tasks and testing its performance on larger models for scaling and optimization.

## Summary

The LLaMA-Berry framework presents an advanced mathematical problem-solving approach aimed at enhancing the reasoning ability of Large Language Models (LLMs). By combining Monte Carlo Tree Search (MCTS) with iterative Self-Refine, the framework optimizes the reasoning path and utilizes a pairwise reward model to globally evaluate different paths. The Self-Refine applied to MCTS (SR-MCTS) technique overcomes inefficiencies of conventional search algorithms by promoting more efficient exploration of solution spaces. Additionally, the Pairwise Preference Reward Model (PPRM) is used to model preferences between solutions, utilizing an Enhanced Borda Count (EBC) method to synthesize these preferences into a global ranking score for improved answers. While the LLaMA-Berry framework has shown strong performance in reasoning tasks, there are challenges in practical applications. Methods like MCTS and Self-Refine have high computational costs, limiting deployment in environments with constrained resources. Rule-based heuristics methods for summarizing solutions have constraints on search performance, prompting the development of a learning-based summarizer for enhanced efficiency. Furthermore, evaluation of the framework has primarily focused on mathematical reasoning benchmarks, necessitating validation in broader domains such as general knowledge and symbolic logic tasks. Future work aims to enhance applicability by evaluating LLaMA-Berry on diverse tasks. Experiments have mainly used small open-source models, warranting investigation into performance on larger models for scaling and optimization. In conclusion, while the LLaMA-Berry framework shows promise in advancing mathematical reasoning capabilities of LLMs, addressing computational costs, expanding evaluation scope, and testing on larger models are crucial steps towards practical implementation and broader applicability across various domains beyond mathematics.