# LongNet: Scaling Transformers to 1,000,000,000 Tokens

**Authors:** Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Furu Wei  
**arXiv:** [2307.02486](https://arxiv.org/abs/2307.02486)  
**Date:** 2023-07-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2307.02486v1/

## Key points

- LongNet is a Transformer variant designed to scale sequence length in large language models.
- Existing methods struggle with computational complexity or model expressivity, limiting maximum sequence length.
- The authors propose dilated attention to exponentially expand the attentive field as token distance increases.
- LongNet can handle sequences longer than 1 billion tokens without sacrificing performance on shorter sequences.
- Advantages of LongNet include linear computation complexity, logarithmic dependency between tokens, and distributed training for extremely long sequences.
- LongNet can seamlessly replace standard attention in existing Transformer-based optimization frameworks.
- Experimental results demonstrate strong performance on both long-sequence modeling and general language tasks.
- This breakthrough enables modeling very long sequences like an entire corpus or the entire Internet as a single sequence.
- Authors of this work: Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Furu Wei.

## Summary

LongNet is a Transformer variant that addresses the need for scaling sequence length in large language models. Existing methods struggle with either computational complexity or model expressivity, limiting the maximum sequence length. To solve this problem, the authors propose dilated attention which exponentially expands the attentive field as the distance between tokens grows. This allows LongNet to scale sequence length to more than 1 billion tokens without sacrificing performance on shorter sequences. LongNet offers several advantages: linear computation complexity and logarithmic dependency between tokens making it efficient and scalable; distributed training for extremely long sequences; seamless replacement of standard attention in existing Transformer-based optimization frameworks. Experimental results show strong performance on both long-sequence modeling and general language tasks. This breakthrough opens up new possibilities for modeling very long sequences such as treating an entire corpus or even the entire Internet as a single sequence. The authors of this work include Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang and Furu Wei.