# Language Modeling Is Compression

**Authors:** Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, Marcus Hutter, Joel Veness  
**arXiv:** [2309.10668](https://arxiv.org/abs/2309.10668)  
**Date:** 2023-09-19  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.10668v1/

## Key points

- Large language models can be viewed as powerful general-purpose compressors due to their predictive capabilities.
- Chinchilla 70B, a large language model trained on text, achieves impressive compression rates of 43.4% on ImageNet patches and 16.4% on LibriSpeech samples.
- Dataset size imposes a limit on model size for optimal compression performance.
- Scaling alone is not a solution for improving compression performance.
- Any compressor, such as gzip, can be used as a conditional generative model by leveraging the equivalence between prediction and compression.
- Tokenization does not necessarily improve compression but allows models to increase information content in context and enhance prediction performance.
- The authors review concepts from information theory related to likelihood maximization and coding distributions.
- Examples of compression-based generation are presented for different types of data including text, audio, and images.
- Performance of gzip is compared with Chinchilla (a large language model) in generating coherent samples based on conditioning contexts.

## Summary

In this work, the authors investigate the lossless compression capabilities of large language models, specifically foundation models trained primarily on text. They argue that these models, due to their impressive predictive capabilities, can also be viewed as powerful general-purpose compressors. The authors make several contributions in their research. Firstly, they empirically explore the compression abilities of foundation models by reviewing how to compress with predictive models using arithmetic coding. They highlight the connection between current language modeling research and compression. The authors demonstrate that large language models are effective general-purpose compressors because of their in-context learning abilities. For example, Chinchilla 70B, a large language model trained on text, achieves impressive compression rates of 43.4% on ImageNet patches and 16.4% on LibriSpeech samples. These rates outperform domain-specific compressors like PNG (58.5%) or FLAC (30.3%), respectively. Additionally, the authors provide a novel perspective on scaling laws in relation to compression performance. They show that dataset size imposes a limit on model size for optimal compression performance and emphasize that scaling alone is not a solution. Furthermore, the authors leverage the equivalence between prediction and compression to use any compressor (such as gzip) as a conditional generative model. The authors also discuss tokenization as a form of pre-compression and its impact on compression performance. They find that tokenization does not necessarily improve compression but allows models to increase information content in context and enhance prediction performance. In terms of background information, the authors review concepts from information theory related to likelihood maximization and coding distributions. To visually illustrate their findings, the authors present examples of compression-based generation for different types of data including text, audio, and images. They compare the performance of gzip with Chinchilla (a large language model) in generating coherent samples based on conditioning contexts.