# Agent Attention: On the Integration of Softmax and Linear Attention

**Authors:** Dongchen Han, Tianzhu Ye, Yizeng Han, Zhuofan Xia, Siyuan Pan, Pengfei Wan, Shiji Song, Gao Huang  
**arXiv:** [2312.08874](https://arxiv.org/abs/2312.08874)  
**Date:** 2023-12-14  
**Summary page:** https://www.summarizepaper.com/en/paper/2312.08874v3/

## Key points

- Authors introduce a novel attention paradigm called Agent Attention
- Agent Attention balances computational efficiency with representation power in Transformers
- Agent Attention is represented as a quadruple $(Q,A,K,V)$ and incorporates an additional set of agent tokens $A$
- By using a smaller number of agent tokens compared to query tokens, Agent Attention offers improved efficiency while maintaining global context modeling capabilities
- Agent Attention is equivalent to a generalized form of linear attention, integrating the strengths of both softmax and linear attention mechanisms
- Extensive experiments across various vision tasks demonstrate the effectiveness of Agent Attention
- In high-resolution scenarios like Stable Diffusion applications, Agent Attention accelerates generation processes and enhances image quality without additional training
- The paper by Han et al. provides valuable insights into how Agent Attention can enhance performance in vision Transformers
- Code for implementing and experimenting with Agent Attention is available on GitHub

## Summary

In their paper titled "Agent Attention: On the Integration of Softmax and Linear Attention," authors Dongchen Han, Tianzhu Ye, Yizeng Han, Zhuofan Xia, Siyuan Pan, Pengfei Wan, Shiji Song, and Gao Huang introduce a novel attention paradigm called Agent Attention. The attention module plays a crucial role in Transformers by providing high expressiveness through global attention mechanisms. However, the computational cost associated with global attention limits its applicability in various scenarios. To address this issue, the authors propose Agent Attention as a way to balance computational efficiency with representation power. represented as a quadruple $(Q,A,K,V)$ and incorporating an additional set of agent tokens $A$, , and  facilitates information aggregation and broadcasting back to the query tokens. By designing the number of agent tokens to be much smaller than query tokens,  offers significantly improved efficiency compared to  attention while maintaining global context modeling capabilities. The authors demonstrate that  is equivalent to a generalized form of linear attention, seamlessly integrating the strengths of both  and linear attention mechanisms. Extensive experiments across various vision tasks such as image classification, object detection, semantic segmentation, and image generation showcase the effectiveness of 

Particularly in high-resolution scenarios like Stable Diffusion applications,< kd > Agent Attention </ kd > accelerates generation processes and enhances image quality without requiring additional training. The paper by Han et al., presented at ECCV 2024 conference,< kd > provides valuable insights into how Agent Attention can enhance performance in vision Transformers.</ kd > The availability of code on GitHub further facilitates implementation and experimentation with this innovative attention paradigm.