Results of the summarizing process for the arXiv paper: 2307.08621v1

In their paper titled "Retentive Network: A Successor to Transformer for Large Language Models," Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei propose a new architecture called Retentive Network (RetNet) as a foundation for large language models. The authors aim to achieve training parallelism, low-cost inference and good performance simultaneously. The researchers start by theoretically deriving the connection between recurrence and attention. They then introduce the retention mechanism for sequence modeling in RetNet. This mechanism supports three computation paradigms: parallel representation; recurrent representation; and chunkwise recurrent representation. The parallel representation allows for training parallelism enabling efficient model training. The recurrent representation enables low-cost $O(1)$ inference improving decoding throughput latency and GPU memory usage without compromising performance. The chunkwise recurrent representation facilitates efficient modeling of long sequences with linear complexity where each chunk is encoded in parallel while summarizing the chunks recursively. Experimental results on language modeling demonstrate that RetNet achieves favorable scaling results in terms of both training and deployment as well as efficient inference with low cost. These intriguing properties position RetNet as a strong successor to the Transformer model for large language models. The authors provide code for implementing RetNet at https://aka.ms/retnet.

• Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei propose a new architecture called Retentive Network (RetNet) for large language models.
• The authors aim to achieve training parallelism, low-cost inference, and good performance simultaneously.
• The retention mechanism in RetNet supports three computation paradigms: parallel representation, recurrent representation, and chunkwise recurrent representation.
• Experimental results show that RetNet achieves favorable scaling results in terms of both training and deployment.
• RetNet enables efficient inference with low cost and improves decoding throughput latency and GPU memory usage without compromising performance.
• RetNet is positioned as a strong successor to the Transformer model for large language models.

Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei created a new architecture called Retentive Network (RetNet) for big language models. They wanted to make it easier to train the models and use them quickly and effectively. RetNet has a special way of remembering information that helps with different types of calculations. The experiments showed that RetNet works well for both training and using the models. It also makes it faster and uses less computer memory. People think that RetNet is a great improvement over the Transformer model for big language models. Definitions - Architecture: The way something is built or organized. - Parallelism: Doing multiple things at the same time. - Inference: Using what you already know to figure out something new. - Computation: Doing math or calculations. - Latency: How long it takes for something to happen after you ask for it.