# Scaling Transformer to 1M tokens and beyond with RMT

**Authors:** Aydar Bulatov, Yuri Kuratov, Mikhail S. Burtsev  
**arXiv:** [2304.11062](https://arxiv.org/abs/2304.11062)  
**Date:** 2023-04-19  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.11062v1/

## Key points

- Technical report titled "Scaling Transformer to 1M tokens and beyond with RMT"
- Innovative approach to extend the context length of BERT
- Recurrent Memory Transformer (RMT) architecture used to increase model's effective context length up to two million tokens
- High memory retrieval accuracy maintained
- Enables storage and processing of both local and global information, allowing for information flow between segments of input sequence through recurrence
- Experiments conducted by Aydar Bulatov, Yuri Kuratov, and Mikhail S. Burtsev demonstrate effectiveness
- Potential to enhance long-term dependency handling in natural language understanding and generation tasks
- Enables large-scale context processing for memory-intensive applications

## Summary

The technical report titled "Scaling Transformer to 1M tokens and beyond with RMT" presents an innovative approach to extend the context length of BERT, a highly effective Transformer-based model in natural language processing. The authors leverage the Recurrent Memory Transformer (RMT) architecture to increase the model's effective context length up to two million tokens while maintaining high memory retrieval accuracy. This method allows for the storage and processing of both local and global information, enabling information flow between segments of the input sequence through recurrence. The experiments conducted by Aydar Bulatov, Yuri Kuratov, and Mikhail S. Burtsev demonstrate that their approach is highly effective and holds significant potential to enhance long-term dependency handling in natural language understanding and generation tasks as well as enable large-scale context processing for memory-intensive applications.