# Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth

**Authors:** Yihe Dong, Jean-Baptiste Cordonnier, Andreas Loukas  
**arXiv:** [2103.03404](https://arxiv.org/abs/2103.03404)  
**Date:** 2021-03-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2103.03404v2/

## Key points

- Authors Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas explore attention-based architectures in machine learning
- Limited understanding of why these architectures are effective
- Proposed novel approach to comprehending self-attention networks by breaking down output into smaller terms involving attention heads across layers
- Without skip connections or MLPs, self-attention's output converges doubly exponentially to a rank-1 matrix
- Introduction of skip connections and MLPs prevents degeneration phenomenon
- Experiments validate identified convergence phenomena across various versions of standard transformer architectures
- Research sheds light on workings of self-attention networks and how architectural elements influence behavior and performance

## Summary

In their paper titled "Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth," authors Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas explore the widespread use of attention-based architectures in machine learning. Despite their popularity, there is still limited understanding of why these architectures are so effective. The authors propose a novel approach to comprehending self-attention networks by breaking down their output into smaller terms involving the operation of attention heads across layers. They reveal that without skip connections or multi-layer perceptrons (MLPs), self-attention's output converges doubly exponentially to a rank-1 matrix. However, the introduction of skip connections and MLPs prevents this degeneration phenomenon from occurring. The experiments conducted by the authors validate these identified convergence phenomena across various versions of standard transformer architectures. This research sheds light on the intricate workings of self-attention networks and highlights how certain architectural elements can influence their behavior and performance. By providing insights into the underlying mechanisms driving the effectiveness of attention-based architectures, this study contributes to advancing our understanding of these widely utilized machine learning models.