# Attention-Only Transformers and Implementing MLPs with Attention Heads

**Authors:** Robert Huben, Valerie Morris  
**arXiv:** [2309.08593](https://arxiv.org/abs/2309.08593)  
**Date:** 2023-09-15  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.08593v1/

## Key points

- Transformer architecture comprises two key sublayers: attention heads and MLPs
- An MLP neuron can be effectively implemented by a masked attention head with an internal dimension of 1
- Increasing the number of attention heads can convert an MLP-and-attention transformer into an attention-only transformer
- Attention heads can encode diverse masking patterns in their weight matrices with minimal error
- Properties of logarithms allow for simplification of requirements related to matrix operations
- Bounding matrix entries using operator norms helps determine optimal values for certain parameters within the model
- Research demonstrates the potential for leveraging attention heads within transformers to emulate functionalities typically associated with MLP layers

## Summary

In their study "Attention-Only Transformers and Implementing MLPs with Attention Heads," researchers Robert Huben and Valerie Morris delve into the transformer architecture commonly used in machine learning models. This architecture comprises two key sublayers: attention heads and MLPs. The researchers demonstrate that an MLP neuron can be effectively implemented by a masked attention head with an internal dimension of 1, given that the activation function of the MLP belongs to a specific restricted class including SiLU, ReLU, and GeLU approximations. They also reveal that increasing the number of attention heads can convert an MLP-and-attention transformer into an attention-only transformer. Furthermore, they establish that attention heads have the ability to encode diverse masking patterns in their weight matrices with minimal error. Expanding on these findings, the researchers explore how properties of logarithms allow for simplification of requirements related to matrix operations. By bounding matrix entries using operator norms, they derive expressions for determining optimal values for certain parameters within the model. Through detailed calculations and examples involving language models like GPT-2, they illustrate how these concepts can be applied in practical scenarios. Overall, this research sheds light on the potential for leveraging attention heads within transformers to emulate functionalities typically associated with MLP layers. It highlights the versatility and efficiency of attention mechanisms in machine learning architectures while offering insights into optimizing model performance through careful consideration of mathematical principles and constraints.