# FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

**Authors:** Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré  
**arXiv:** [2205.14135](https://arxiv.org/abs/2205.14135)  
**Date:** 2022-05-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2205.14135v1/

## Key points

- FlashAttention is a new attention algorithm that addresses the issue of slow and memory-intensive Transformers on long sequences.
- The problem arises due to the quadratic time and memory complexity of self-attention with increasing sequence length.
- FlashAttention proposes an IO-aware exact attention algorithm that accounts for reads and writes between levels of GPU memory, which is missing in existing attention algorithms.
- FlashAttention uses tiling to reduce the number of memory reads/writes between GPU high bandwidth memory (HBM) and GPU on-chip SRAM.
- The authors demonstrate that FlashAttention trains Transformers faster than existing baselines, achieving a 15% end-to-end wall-clock speedup on BERT-large (seq. length 512), 3x speedup on GPT-2 (seq. length 1K), and 2.4x speedup on long-range arena (seq. length 1K-4K).
- FlashAttention enables longer context in Transformers, leading to higher quality models with better perplexity on GPT-2 by 0.7 points and a lift of 6.4 points on long-document classification.
- FlashAttention achieves entirely new capabilities by enabling the first Transformers to achieve better-than-chance performance on the Path-X challenge (seq. length 16K, accuracy of 61.4%) and Path-256 (seq."

## Summary

FlashAttention is a new attention algorithm that addresses the issue of slow and memory-intensive Transformers on long sequences. This problem arises due to the quadratic time and memory complexity of self-attention with increasing sequence length. While approximate attention methods have attempted to tackle this issue, they often compromise model quality without achieving significant speedup. The authors propose an IO-aware exact attention algorithm that accounts for reads and writes between levels of GPU memory, which is missing in existing attention algorithms. FlashAttention uses tiling to reduce the number of memory reads/writes between GPU high bandwidth memory (HBM) and GPU on-chip SRAM. The IO complexity analysis shows that FlashAttention requires fewer HBM accesses than standard attention and is optimal for a range of SRAM sizes. Additionally, FlashAttention is extended to block-sparse attention, resulting in an approximate attention algorithm that outperforms any existing method in terms of speed. The authors demonstrate that FlashAttention trains Transformers faster than existing baselines, achieving a 15% end-to-end wall-clock speedup on BERT-large (seq. length 512) compared to the MLPerf 1.1 training speed record, 3x speedup on GPT-2 (seq. length 1K), and 2.4x speedup on long-range arena (seq. length 1K-4K). Moreover, FlashAttention enables longer context in Transformers, leading to higher quality models with better perplexity on GPT-2 by 0.7 points and a lift of 6.4 points on long-document classification. FlashAttention also achieves entirely new capabilities by enabling the first Transformers to achieve better-than-chance performance on the Path-X challenge (seq. length 16K, accuracy of 61.4%) and Path-256 (seq.