# RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold

**Authors:** Amrith Setlur, Saurabh Garg, Xinyang Geng, Naman Garg, Virginia Smith, Aviral Kumar  
**arXiv:** [2406.14532](https://arxiv.org/abs/2406.14532)  
**Date:** 2024-06-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2406.14532v1/

## Key points

- Authors explore training language models on model-generated synthetic data for math reasoning tasks
- Sampling more correct solutions from the finetuned learner and fine-tuning on self-generated data doubles efficiency in solving synthetic problems
- Constructing negative responses to mitigate potential pitfalls of training on model-generated positives leads to consistent gains
- Comparison between positive synthetic data from larger models like GPT-4 and Gemini 1.5 Pro with self-generated positive data highlights benefits of learning generalizable features
- Training on both positive and negative synthetic data enhances reasoning abilities, mitigates biases, and prevents undesirable memorization

## Summary

The authors of this paper delve into the realm of training language models on model-generated synthetic data for math reasoning tasks. They begin by exploring the effectiveness of finetuning LLMs on synthetic correct or positive problem-solution pairs generated by proficient models. However, they make a groundbreaking discovery that sampling more correct solutions from the finetuned learner itself and subsequently fine-tuning on this self-generated data results in a doubling of efficiency when solving the same synthetic problems. The study also uncovers potential pitfalls of training on model-generated positives and introduces negative responses to mitigate these issues. By constructing negatives that allow for appropriate recovery of each intermediate step's utility or advantage, they achieve consistent gains over using only positive data. Furthermore, the paper delves into related works in the field and compares performance scaling with positive synthetic data from larger models like GPT-4 and Gemini 1.5 Pro with self-generated positive data. Additionally, the study explores the benefits and nuances of negative synthetic data in math reasoning tasks and establishes an equivalence between preference optimization and advantage-weighted reinforcement learning through a framework of offline preference optimization. Overall, this comprehensive analysis sheds light on how training language models on both positive and negative synthetic data can enhance reasoning abilities while mitigating biases and spurious correlations often associated with solely relying on positive responses. : Training language models on model-generated synthetic data for mathematical problem-solving. : Generated by proficient models to improve performance gains when finetuning LLMs. : Trained on both positive and negative synthetic data to enhance reasoning abilities. : Introduced to mitigate potential pitfalls of training solely on model-generated positives. : Used in comparison with self-generated positive data to highlight the benefits of learning generalizable features and preventing undesirable memorization.