# Agent Attention: On the Integration of Softmax and Linear Attention

**Authors:** Dongchen Han, Tianzhu Ye, Yizeng Han, Zhuofan Xia, Shiji Song, Gao Huang  
**arXiv:** [2312.08874](https://arxiv.org/abs/2312.08874)  
**Date:** 2023-12-14  
**Summary page:** https://www.summarizepaper.com/en/paper/2312.08874v2/

## Key points

- Agent Attention is a new attention paradigm in Transformers that balances computational efficiency and representation power.
- It introduces agent tokens that act as agents for query tokens, aggregating information from key and value tokens and broadcasting it back to the query tokens.
- By designing the number of agent tokens to be smaller than the number of query tokens, Agent Attention achieves higher efficiency compared to Softmax attention while still preserving global context modeling capability.
- Agent Attention is equivalent to a generalized form of linear attention, seamlessly integrating both Softmax attention and highly efficient linear attention.
- Extensive experiments validate the effectiveness of Agent Attention in various vision tasks such as image classification, object detection, semantic segmentation, and image generation.
- In high-resolution scenarios, Agent Attention outperforms other methods due to its linear attention nature.
- When applied to Stable Diffusion for image generation without additional training, Agent Attention accelerates generation and substantially improves image quality.
- The code for implementing Agent Attention is available on GitHub.

## Summary

, , , , 

The attention module is a crucial component in Transformers, but its high computational cost limits its practicality in many scenarios. To address this issue, the authors propose a new attention paradigm called Agent Attention. This approach strikes a balance between computational efficiency and representation power by introducing agent tokens into the conventional attention module. These tokens act as agents for the query tokens, aggregating information from key and value tokens and broadcasting it back to the query tokens. By designing the number of agent tokens to be smaller than the number of query tokens, Agent Attention achieves higher efficiency compared to Softmax attention while still preserving global context modeling capability. Interestingly, the authors demonstrate that Agent Attention is equivalent to a generalized form of linear attention, seamlessly integrating both Softmax attention and highly efficient linear attention. Extensive experiments with various vision Transformers and tasks such as image classification, object detection, semantic segmentation, and image generation validate the effectiveness of Agent Attention. Notably, in high-resolution scenarios, Agent Attention outperforms other methods due to its linear attention nature. Furthermore, when applied to Stable Diffusion for image generation without additional training, Agent Attention accelerates generation and substantially improves image quality. The code for implementing Agent Attention is available on GitHub. In summary,<br>
this paper introduces Agent Attention as a novel attention paradigm that balances computational efficiency and representation power in Transformers.<br>
Its integration of Softmax and linear attention enables efficient global context modeling across diverse vision tasks with remarkable performance in high-resolution scenarios.