# Large Language Monkeys: Scaling Inference Compute with Repeated Sampling

**Authors:** Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, Azalia Mirhoseini  
**arXiv:** [2407.21787](https://arxiv.org/abs/2407.21787)  
**Date:** 2024-07-31  
**Summary page:** https://www.summarizepaper.com/en/paper/2407.21787v1/

## Key points

- Scaling compute has enhanced language model training capabilities
- Inference compute as a dimension for scaling by increasing generated samples
- Coverage scales with number of samples, leading to improved performance in coding and formal proofs
- Repeated sampling increased problem-solving success rates significantly
- Amplifying more affordable models with multiple samples can be cost-effective and efficient
- Log-linear relationship between coverage and number of samples identified
- Identifying correct samples crucial for domains lacking automatic verifiers
- Traditional methods like majority voting or reward models plateaued beyond several hundred samples
- Cautionary tales shared regarding inconsistencies in test suites and limitations in verification tools
- Scaling inference compute through repeated sampling can lead to significant improvements in coverage across tasks and models

## Summary

In the realm of language model training, scaling compute has significantly enhanced the capabilities of models. However, when it comes to inference, we often limit the amount of compute to just one attempt per problem. In this study, we delve into the concept of inference compute as another dimension for scaling by increasing the number of generated samples. Across various tasks and models, it is observed that coverage - which refers to the fraction of problems solved by any attempt - scales with the number of samples over four orders of magnitude. This increase in coverage directly translates into improved performance in domains like coding and formal proofs where answers can be automatically verified. For instance, when applying repeated sampling to SWE-bench Lite, it was found that the fraction of issues solved with DeepSeek-V2-Coder-Instruct increased from 15.9% with one sample to 56% with 250 samples, surpassing the single-attempt state-of-the-art success rate of 43%. Moreover, utilizing current API pricing revealed that amplifying the more affordable DeepSeek model with five samples proved to be more cost-effective and efficient in solving problems compared to paying a premium for one sample from more advanced models like GPT-4o or Claude 3.5 Sonnet. Interestingly, a log-linear relationship between coverage and the number of samples was identified across multiple experiments, suggesting the existence of inference-time scaling laws. Furthermore, it was noted that while identifying correct samples out of numerous generations remains crucial for domains lacking automatic verifiers. from GSM8K and MATH datasets using Llama-3 models resulted in coverage exceeding 95% with 10,000 samples. However, traditional methods such as majority voting or reward models plateaued beyond several hundred samples and did not fully scale with the sample budget. Additionally, cautionary tales were shared regarding software development tasks and verification tools. were identified in SWE-bench Lite where inconsistencies in test suites led to false positives/negatives even on ground-truth solutions. Similarly, false negatives were encountered in CodeContests due to limitations in existing verification tools. Overall, this study highlights that scaling inference compute through repeated sampling can lead to significant improvements in coverage across various tasks and models. It demonstrates how amplifying weaker models with multiple samples can sometimes outperform single attempts from more powerful models. The findings underscore the importance of robust verifiers and highlight areas for further research in improving decision-making processes based on sampled outputs.