# Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling

**Authors:** Hritik Bansal, Arian Hosseini, Rishabh Agarwal, Vinh Q. Tran, Mehran Kazemi  
**arXiv:** [2408.16737](https://arxiv.org/abs/2408.16737)  
**Date:** 2024-08-29  
**Summary page:** https://www.summarizepaper.com/en/paper/2408.16737v2/

## Key points

- Using high-quality synthetic data from language models (LMs) is a popular strategy in natural language processing for improving reasoning capabilities.
- The study by Hritik Bansal et al. questions the compute optimality of using synthetic data from robust but computationally expensive (SE) models versus weaker but more cost-effective (WC) models.
- Three key metrics - coverage, diversity, and false positive rate - are used to evaluate the generated data, showing that WC-generated data may have higher coverage and diversity but also higher false positive rates compared to SE-generated data.
- Experiments show that models trained on WC-generated data consistently outperform those trained on SE-generated data across multiple benchmarks and scenarios like knowledge distillation and self-improvement.
- Results challenge the conventional practice of relying solely on SE models for synthetic data generation, suggesting that leveraging WC models may be a more compute-optimal approach for enhancing reasoning abilities in advanced LM systems.

## Summary

In the realm of natural language processing, using high-quality synthetic data generated from strong language models (LMs) has become a popular strategy to improve reasoning capabilities. However, a recent study by Hritik Bansal et al. titled "Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling" questions whether this approach is truly compute-optimal when considering a fixed inference budget. The research explores the trade-offs between using synthetic data from a more robust but computationally expensive (SE) model versus a weaker but more cost-effective (WC) model. Three key metrics - coverage, diversity, and false positive rate - are used to evaluate the generated data. Surprisingly, the findings show that while WC-generated data may have higher coverage and diversity compared to SE-generated data, they also tend to have higher false positive rates. To further investigate this phenomenon, experiments are conducted where LMs are fine-tuned using data from both SE and WC models in various scenarios such as knowledge distillation and self-improvement. A unique weak-to-strong improvement setup is also explored where a less powerful LM imparts reasoning skills to a stronger LM. Remarkably, the results consistently demonstrate that models trained on WC-generated data outperform those trained on SE-generated data across multiple benchmarks and different combinations of WC and SE models. These groundbreaking findings challenge the conventional practice of relying solely on SE models for synthetic data generation in LM training and suggest that leveraging WC models may be a more compute-optimal approach for enhancing reasoning abilities in advanced LM systems. This research sheds new light on optimizing computational resources in training sophisticated language models and opens up avenues for further exploration in this domain.