# Scaling Relationship on Learning Mathematical Reasoning with Large Language Models

**Authors:** Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, Chang Zhou  
**arXiv:** [2308.01825](https://arxiv.org/abs/2308.01825)  
**Date:** 2023-08-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2308.01825v1/

## Key points

- Study titled "Scaling Relationship on Learning Mathematical Reasoning with Large Language Models" by Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou
- Focus on mathematical reasoning for large language models (LLMs) and the relationship between LLM capacity and performance
- Pre-training loss as a reliable indicator of model performance compared to the number of parameters
- Log-linear correlation between data volume and model proficiency through supervised fine-tuning (SFT)
- Diminishing returns for superior models with larger supervised datasets
- Introduction of Rejection sampling Fine-Tuning (RFT) to enhance model performance without human intervention
- RFT shows significant improvements in mathematical reasoning capabilities for LLMs by incorporating diverse reasoning pathways in augmented samples
- More pronounced enhancements observed for less proficient LLMs with RFT
- Remarkable results achieved with LLaMA-7B reaching an accuracy rate of 49.3% using RFT compared to 35.9% with traditional SFT
- Emphasis on effective strategies for enhancing mathematical reasoning in large language models through innovative techniques like RFT and consideration of pre-training loss alongside other factors for evaluating model performance

## Summary

In the study titled "Scaling Relationship on Learning Mathematical Reasoning with Large Language Models," authors Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou delve into the complexities of mathematical reasoning for large language models (LLMs) and explore the relationship between LLM capacity and mathematical reasoning performance. The researchers focus on how factors such as pre-training loss, supervised data quantity, and augmented data impact the reasoning abilities of supervised LLMs. Through their investigation, the team discovers that pre-training loss serves as a more reliable indicator of model performance compared to the sheer number of parameters in the model. By employing supervised fine-tuning (SFT) with varying amounts of labeled data, they establish a log-linear correlation between data volume and model proficiency. Interestingly, they observe that superior models exhibit diminishing returns when exposed to larger supervised datasets. To enhance model performance without additional human intervention, the researchers propose a novel approach called Rejection sampling Fine-Tuning (RFT). This method leverages supervised models to generate and compile accurate reasoning paths as augmented fine-tuning datasets. Their experiments reveal that RFT yields significant improvements in mathematical reasoning capabilities for LLMs by incorporating diverse reasoning pathways in augmented samples. Notably, RFT demonstrates more pronounced enhancements for less proficient LLMs. Moreover, by amalgamating rejection samples from multiple models, the team achieves remarkable results with LLaMA-7B reaching an accuracy rate of 49.3%. This surpasses the accuracy attained through traditional supervised fine-tuning (SFT), which stood at 35.9%. Overall, this research sheds light on effective strategies for enhancing mathematical reasoning in large language models through innovative techniques like RFT and highlights the importance of considering pre-training loss alongside other factors when evaluating model performance.