# Will we run out of data? Limits of LLM scaling based on human-generated data

**Authors:** Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, Marius Hobbhahn  
**arXiv:** [2211.04325](https://arxiv.org/abs/2211.04325)  
**Date:** 2022-10-26  
**Summary page:** https://www.summarizepaper.com/en/paper/2211.04325v2/

## Key points

- Authors examine potential constraints on Large Language Model (LLM) scaling due to availability of public human-generated text data
- Predict LLM models will be trained on datasets equal in size to available stock of public human text data between 2026 and 2032
- Trend may exhaust supply of public human text data, hindering further scaling beyond this decade
- Strategies explored to overcome constraint include synthetic data generation, transfer learning from data-rich domains, and utilizing non-public sources
- Concerns raised about high-quality training data becoming bottleneck for machine learning progress
- Solutions suggested include repeating data, adding more code data, multi-epoch training, using verification processes as training signals
- Exploration of training models on synthetic feedback and non-text modalities like images to overcome limitations posed by finite supplies of public human text data

## Summary

In this paper, the authors examine the potential constraints on Large Language Model (LLM) scaling caused by the availability of public human-generated text data. They predict that LLM models will soon be trained on datasets equal in size to the available stock of public human text data between 2026 and 2032. The authors argue that this trend may exhaust the supply of public human text data and hinder further scaling beyond this decade. To support their conclusion, they develop a model of demand for training data and production of public human text data. The paper also explores strategies such as synthetic data generation, transfer learning from data-rich domains, and utilizing non-public sources to overcome this constraint. It also reviews related work on internet data stock estimates and studies on mitigating data scarcity in machine learning. Researchers have expressed concerns about high-quality training data becoming a bottleneck for machine learning progress and suggest solutions such as repeating data, adding more code data, multi-epoch training, and using verification processes as training signals. Additionally, researchers have explored training models on synthetic feedback and non-text modalities like images to overcome limitations posed by finite supplies of public human text data. The paper includes rough estimates of the stock of different modalities such as video and image datasets. Overall, this study highlights the importance of investigating limitations imposed by finite supplies of public human text data and explores various strategies to support further progress in language modeling beyond current constraints.