# Is Attention All What You Need? -- An Empirical Investigation on Convolution-Based Active Memory and Self-Attention

**Authors:** Thomas Dowdell, Hongyu Zhang  
**arXiv:** [1912.11959](https://arxiv.org/abs/1912.11959)  
**Date:** 2019-12-27  
**Summary page:** https://www.summarizepaper.com/en/paper/1912.11959v2/

## Key points

- Study by Thomas Dowdell and Hongyu Zhang on active-memory mechanisms vs. self-attention in Transformer models
- Focus on effectiveness of active-memory mechanisms in language modeling and algorithmic tasks
- Active-memory alone can achieve similar results to self-attention in language modeling, but combining both often leads to optimal performance
- Active-memory mechanisms outperform self-attention and combination for specific algorithmic tasks
- All models perform well on Not function task due to efficient analysis of input-output dependencies
- Self-attention excels in long-range dependency capabilities compared to active-memory for tasks like Remember
- Using either mechanism alone is better than a combination for the Remember function task, indicating a complex interplay between them
- Study highlights potential of active-memory mechanisms as alternative or complement to traditional self-attention in Transformer models

## Summary

In their study "Is Attention All What You Need? - An Empirical Investigation on Convolution-Based Active Memory and Self-Attention," researchers Thomas Dowdell and Hongyu Zhang explore the effectiveness of active-memory mechanisms as a replacement for self-attention in Transformer models. The main focus is on whether various active-memory mechanisms can achieve comparable results to self-attention in language modeling and algorithmic tasks. The experiments reveal that while active-memory alone can achieve similar results to self-attention in language modeling, optimal performance is often achieved by combining both mechanisms. Interestingly, for specific algorithmic tasks, active-memory mechanisms outperform both self-attention and a combination of the two. Notably, all models perform well on the Not function task due to their ability to efficiently analyze input-output dependencies at each time-step. However, the self-attention mechanism shows superior long-range dependency capabilities compared to active-memory mechanisms in tasks like Remember. Surprisingly, using either mechanism alone outperforms their combination for the Remember function task, suggesting a complex interplay between them that requires further investigation. Overall, this study highlights the potential of active-memory mechanisms as an alternative or complement to traditional self-attention in Transformer models. By understanding how these mechanisms interact and perform across different tasks, researchers can optimize model performance for various applications in natural language processing and beyond.