# Improving language models by retrieving from trillions of tokens

**Authors:** Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock  
**arXiv:** [2112.04426](https://arxiv.org/abs/2112.04426)  
**Date:** 2021-12-08  
**Summary page:** https://www.summarizepaper.com/en/paper/2112.04426v1/

## Key points

- Language models have made remarkable progress in natural language processing tasks but still have limitations.
- The proposed Retrieval-Enhanced Transformer (RETRO) enhances auto-regressive language models by conditioning on document chunks retrieved from a large corpus based on local similarity with preceding tokens.
- RETRO achieves comparable performance to GPT-3 and Jurassic-1 on the Pile dataset despite using 25 times fewer parameters.
- RETRO combines a frozen Bert retriever, a differentiable encoder, and a chunked cross-attention mechanism to predict tokens based on an order of magnitude more data than what is typically consumed during training.
- RETRO's performance translates to downstream knowledge-intensive tasks such as question answering after fine-tuning.
- The model can be trained from scratch or rapidly retrofit pre-trained transformers with retrieval and still achieve good performance.
- Improvements do not diminish as they scale the models or increase the number of retrieved chunks.
- Jurassic-1 outperforms their baseline on all datasets except for books, likely due to the inclusion of books in their training data.
- This work opens up new avenues for improving language models through explicit memory at unprecedented scale by retrieving information from trillions of tokens.

## Summary

In recent years, language models have shown remarkable progress in various natural language processing tasks. However, these models still suffer from limitations such as lack of explicit memory and inability to handle long-range dependencies. To address these issues, a team of researchers from DeepMind has proposed a Retrieval-Enhanced Transformer (RETRO) that enhances auto-regressive language models by conditioning on document chunks retrieved from a large corpus based on local similarity with preceding tokens. With a database of 2 trillion tokens, RETRO achieves comparable performance to GPT-3 and Jurassic-1 on the Pile dataset despite using 25 times fewer parameters. The model combines a frozen Bert retriever, a differentiable encoder, and a chunked cross-attention mechanism to predict tokens based on an order of magnitude more data than what is typically consumed during training. After fine-tuning, RETRO's performance translates to downstream knowledge-intensive tasks such as question answering. The researchers train RETRO from scratch but can also rapidly retrofit pre-trained transformers with retrieval and still achieve good performance. They evaluate the model's performance by scaling both the model size and the retrieval database at evaluation. They observe that improvements do not diminish as they scale the models or increase the number of retrieved chunks. The team evaluates their 7 billion parameter RETRO model on the Pile test sets and compares it against Jurassic-1 and Gopher models. They find that Jurassic-1 outperforms their baseline on all datasets except for books, likely due to the inclusion of books in their training data. On Curation Corpus, they concatenate the article, "TL;DR:" string, and summary but only evaluate bits-per-byte (bpb) on the summary. Overall, this work opens up new avenues for improving language models through explicit memory at unprecedented scale by retrieving information from trillions of tokens.