# Are Your LLMs Capable of Stable Reasoning?

**Authors:** Junnan Liu, Hongwei Liu, Linchen Xiao, Ziyi Wang, Kuikun Liu, Songyang Gao, Wenwei Zhang, Songyang Zhang, Kai Chen  
**arXiv:** [2412.13147](https://arxiv.org/abs/2412.13147)  
**Date:** 2024-12-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2412.13147v2/

## Key points

- Authors address the gap between benchmark performances and real-world applications in Large Language Models (LLMs)
- Current evaluation protocols and metrics fail to fully capture the diverse capabilities of LLMs, especially in complex reasoning tasks
- Introduce G-Pass@k as a novel evaluation metric for continuous assessment of model performance and stability
- Present LiveMathBench as a dynamic benchmark for challenging mathematical problems to minimize data leakage risks during evaluation
- Extensive experiments using G-Pass@k on cutting-edge LLMs with LiveMathBench provide insights into maximum capabilities and operational consistency of these models
- Findings highlight room for improvement in LLMs' "realistic" reasoning abilities and emphasize the need for more robust evaluation methods

## Summary

In their paper titled "Are Your LLMs Capable of Stable Reasoning? ", authors Junnan Liu, Hongwei Liu, Linchen Xiao, Ziyi Wang, Kuikun Liu, Songyang Gao, Wenwei Zhang, Songyang Zhang and Kai Chen address the significant gap between benchmark performances and real-world applications in Large Language Models (LLMs). They argue that this disconnect is primarily due to current evaluation protocols and metrics failing to fully capture the diverse capabilities of LLMs. This is especially evident in complex reasoning tasks where accuracy and consistency are paramount. To bridge this divide, the authors introduce two key contributions. Firstly, they propose G-Pass@k - a novel evaluation metric that offers a continuous assessment of model performance across multiple sampling attempts. This metric not only quantifies the peak performance potential of LLMs but also evaluates their stability over various scenarios. Secondly, they present LiveMathBench - a dynamic benchmark consisting of challenging mathematical problems designed to minimize data leakage risks during evaluation. Through extensive experiments utilizing G-Pass@k on cutting-edge LLMs with LiveMathBench, the authors provide comprehensive insights into both the maximum capabilities and operational consistency of these models. Their findings underscore substantial room for improvement in LLMs' "realistic" reasoning abilities and emphasize the necessity for more robust evaluation methods in assessing these advanced language models. The detailed results and benchmark information can be accessed at: https://github.com/open-compass/GPassK.