# Generating Long Sequences with Sparse Transformers

**Authors:** Rewon Child, Scott Gray, Alec Radford, Ilya Sutskever  
**arXiv:** [1904.10509](https://arxiv.org/abs/1904.10509)  
**Date:** 2019-04-23  
**Summary page:** https://www.summarizepaper.com/en/paper/1904.10509v1/

## Key points

- Authors address time and memory requirements in Transformers
- Proposed sparse factorizations of attention matrix to reduce complexity
- Introduced architectural variations and initialization techniques for training deeper networks
- Suggested recomputing attention matrices and fast attention kernels for efficient training
- Sparse Transformers capable of modeling sequences with tens of thousands of timesteps using hundreds of layers
- Applied same architecture to model images, audio, and text achieving state-of-the-art performance
- Demonstrated ability to generate unconditional samples with global coherence and diversity
- Self-attention can potentially model sequences exceeding one million in length
- Comprehensive approach to addressing limitations of traditional Transformers by introducing sparse factorizations and modifications

## Summary

In the paper "Generating Long Sequences with Sparse Transformers," authors Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever address the issue of time and memory requirements in Transformers, which grow quadratically with sequence length. They propose sparse factorizations of the attention matrix to reduce this complexity to O(n√n). Additionally, they introduce architectural variations and initialization techniques to train deeper networks. The authors also suggest recomputing attention matrices to save memory and fast attention kernels for efficient training. These modifications result in what they call Sparse Transformers, capable of modeling sequences with tens of thousands of timesteps using hundreds of layers. The same architecture is applied to model images, audio, and text from raw bytes, achieving state-of-the-art performance in density modeling for Enwik8, CIFAR-10, and ImageNet-64 datasets. The authors demonstrate the ability of Sparse Transformers to generate unconditional samples that exhibit global coherence and great diversity. Furthermore, they show that self-attention can potentially be used to model sequences with lengths exceeding one million. Overall, this paper presents a comprehensive approach to addressing the limitations of traditional Transformers by introducing sparse factorizations and other modifications that enable more efficient modeling of long sequences while maintaining high performance across various domains.