# Scaling Laws for Neural Language Models

**Authors:** Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei  
**arXiv:** [2001.08361](https://arxiv.org/abs/2001.08361)  
**Date:** 2020-01-23  
**Summary page:** https://www.summarizepaper.com/en/paper/2001.08361v1/

## Key points

- Loss scales as a power-law with respect to model size, dataset size, and compute used for training
- Architectural details such as network width and depth have minimal effects on language model performance
- Overfitting depends on both model and dataset size
- Training speed is related to model size, allowing for optimal allocation of compute budget
- Larger models are more sample-efficient compared to smaller ones
- Compute-efficient training involves training large models on a modest amount of data and stopping before convergence
- Model size and dataset size are important factors in optimizing compute efficiency in language model training

## Summary

In the study titled "Scaling Laws for Neural Language Models," authors Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu and Dario Amodei investigate the empirical scaling laws that govern language model performance on the cross-entropy loss. They find that the loss scales as a power-law with respect to model size, dataset size and compute used for training; these scaling trends span more than seven orders of magnitude. The authors also explore the impact of architectural details such as network width and depth on language model performance and discover that within a wide range of values these details have minimal effects on the loss. This finding suggests that other factors play a more significant role in determining language model performance. Through their research they establish simple equations that describe how overfitting depends on both model and dataset size; additionally they uncover relationships between training speed and model size which enable them to determine the optimal allocation of a fixed compute budget. One key finding is that larger models are significantly more sample-efficient compared to smaller ones; consequently achieving compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence. Overall this study provides valuable insights into the scaling laws governing language model performance; the findings highlight the importance of considering factors such as model size and dataset size when optimizing compute efficiency in language model training.