# Linformer: Self-Attention with Linear Complexity

**Authors:** Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, Hao Ma  
**arXiv:** [2006.04768](https://arxiv.org/abs/2006.04768)  
**Date:** 2020-06-08  
**Summary page:** https://www.summarizepaper.com/en/paper/2006.04768v1/

## Key points

- Large transformer models are successful in NLP applications but can be expensive to train and deploy for long sequences due to the standard self-attention mechanism of the Transformer.
- The self-attention mechanism has a time and space complexity of $O(n^2)$ with respect to sequence length.
- A new self-attention mechanism called Linformer is proposed that approximates the original one using a low-rank matrix, reducing overall self-attention complexity from $O(n^2)$ to $O(n)$ in both time and space.
- The Linformer performs on par with standard Transformer models while being much more memory and time-efficient.
- Experiments conducted on several NLP tasks such as machine translation and language modeling show that the Linformer achieves comparable or better performance while requiring significantly less memory usage and computation time compared to other state-of-the-art models such as BERT and RoBERTa.
- The authors provide insights into why their approach works by analyzing how different components of their method contribute to its effectiveness, and explore how varying hyperparameters affects performance.
- Overall, this paper presents an innovative solution to address one of the major challenges in deploying large transformer models for NLP tasks.

## Summary

The success of large transformer models in natural language processing (NLP) applications is undeniable. However, their training and deployment can be prohibitively expensive for long sequences due to the standard self-attention mechanism of the Transformer, which has a time and space complexity of $O(n^2)$ with respect to sequence length. In this paper titled "Linformer: Self-Attention with Linear Complexity," Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma propose a new self-attention mechanism that approximates the original one using a low-rank matrix. This approximation reduces the overall self-attention complexity from $O(n^2)$ to $O(n)$ in both time and space. The resulting linear transformer model is called the Linformer and it performs on par with standard Transformer models while being much more memory- and time-efficient. The authors demonstrate that their proposed method is effective by conducting experiments on several NLP tasks such as machine translation and language modeling. They compare the performance of the Linformer against other state-of-the-art models such as BERT and RoBERTa. The results show that the Linformer achieves comparable or better performance while requiring significantly less memory usage and computation time. Furthermore, they provide insights into why their approach works by analyzing how different components of their method contribute to its effectiveness. They also explore how varying hyperparameters affects performance. Overall, this paper presents an innovative solution to address one of the major challenges in deploying large transformer models for NLP tasks. The Linformer's ability to reduce memory usage and computation time without sacrificing performance makes it a promising candidate for practical applications where efficiency is crucial.