# Retentive Network: A Successor to Transformer for Large Language Models

**Authors:** Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, Furu Wei  
**arXiv:** [2307.08621](https://arxiv.org/abs/2307.08621)  
**Date:** 2023-07-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2307.08621v1/

## Key points

- Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei propose a new architecture called Retentive Network (RetNet) for large language models.
- The authors aim to achieve training parallelism, low-cost inference, and good performance simultaneously.
- The retention mechanism in RetNet supports three computation paradigms: parallel representation, recurrent representation, and chunkwise recurrent representation.
- Experimental results show that RetNet achieves favorable scaling results in terms of both training and deployment.
- RetNet enables efficient inference with low cost and improves decoding throughput latency and GPU memory usage without compromising performance.
- RetNet is positioned as a strong successor to the Transformer model for large language models.

## Summary

In their paper titled "Retentive Network: A Successor to Transformer for Large Language Models," Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei propose a new architecture called Retentive Network (RetNet) as a foundation for large language models. The authors aim to achieve training parallelism, low-cost inference and good performance simultaneously. The researchers start by theoretically deriving the connection between recurrence and attention. They then introduce the retention mechanism for sequence modeling in RetNet. This mechanism supports three computation paradigms: parallel representation; recurrent representation; and chunkwise recurrent representation. The parallel representation allows for training parallelism enabling efficient model training. The recurrent representation enables low-cost $O(1)$ inference improving decoding throughput latency and GPU memory usage without compromising performance. The chunkwise recurrent representation facilitates efficient modeling of long sequences with linear complexity where each chunk is encoded in parallel while summarizing the chunks recursively. Experimental results on language modeling demonstrate that RetNet achieves favorable scaling results in terms of both training and deployment as well as efficient inference with low cost. These intriguing properties position RetNet as a strong successor to the Transformer model for large language models. The authors provide code for implementing RetNet at https://aka.ms/retnet.