# Stabilizing Transformer Training by Preventing Attention Entropy Collapse

**Authors:** Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Dan Busbridge, Jason Ramapuram, Yizhe Zhang, Jiatao Gu, Josh Susskind  
**arXiv:** [2303.06296](https://arxiv.org/abs/2303.06296)  
**Date:** 2023-03-11  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.06296v1/

## Key points

- The authors focus on stabilizing Transformer training by preventing attention entropy collapse
- Tracking attention entropy for each attention head reveals a consistent trend across various architectures and tasks
- Low attention entropy is linked to high training instability characterized by oscillating loss or divergence (entropy collapse)
- The proposed solution, $\sigma$Reparam, involves reparameterizing linear layers using spectral normalization and an additional learned scalar
- Extensive experimentation demonstrates the efficacy of $\sigma$Reparam in preventing entropy collapse within attention layers
- The reparameterization technique promotes stable training and robustness in handling varying hyperparameters
- A rigorous lower bound for attention entropy diminishes exponentially with the spectral norm of attention logits
- Noteworthy results include competitive performance without warmup procedures, weight decay techniques, layer normalization methods, or adaptive optimizers

## Summary

, , , , 


In their work titled "Stabilizing Transformer Training by Preventing Attention Entropy Collapse," the authors delve into the crucial aspect of training stability in Transformers. By closely examining the training dynamics of Transformers and focusing on the evolution of attention layers, they aim to shed light on the factors influencing model sharpness. Through tracking the attention entropy for each attention head throughout training, which serves as an indicator of model focus and concentration, they uncover a consistent trend across various architectures and tasks. The study reveals that low attention entropy is often linked to high training instability characterized by oscillating loss or divergence. This phenomenon is termed "entropy collapse," signifying excessively concentrated attention scores within the model. To address this issue effectively, the authors propose a solution called $\sigma$Reparam. This innovative approach involves reparameterizing all linear layers using spectral normalization alongside an additional learned scalar. Through extensive experimentation across diverse tasks such as image classification, image self-supervised learning, machine translation, automatic speech recognition, and language modeling tasks spanning different Transformer architectures, the authors demonstrate the efficacy of $\sigma$Reparam in preventing entropy collapse within attention layers. The proposed reparameterization technique not only promotes more stable training but also showcases robustness in handling varying hyperparameters. Furthermore, the authors establish a rigorous lower bound for attention entropy that diminishes exponentially with the spectral norm of attention logits. This finding provides further justification for adopting their approach. Noteworthy results include achieving competitive performance without relying on warmup procedures, weight decay techniques, layer normalization methods, or adaptive optimizers in scenarios such as training Vision Transformers and deep architectures for machine translation and speech recognition tasks. Overall, this research contributes significantly to enhancing stability and robustness in Transformer training processes while offering valuable insights into mitigating issues related to attention entropy collapse through innovative reparameterization strategies.