# Attention is not not Explanation

**Authors:** Sarah Wiegreffe, Yuval Pinter  
**arXiv:** [1908.04626](https://arxiv.org/abs/1908.04626)  
**Date:** 2019-08-13  
**Summary page:** https://www.summarizepaper.com/en/paper/1908.04626v2/

## Key points

- Role of attention mechanisms in NLP systems, specifically in RNN models
- Challenge to the claim that "Attention is not Explanation"
- Four alternative tests proposed to determine the use of attention as an explanation:
- Simple uniform-weights baseline
- Variance calibration based on multiple random seed runs
- Diagnostic framework using frozen weights from pretrained models
- End-to-end adversarial attention training protocol
- Meaningful interpretation of attention mechanisms in RNN models
- Evidence suggesting that prior work does not disprove the usefulness of attention mechanisms for explainability
- Different notions of transparency, explainability, and interpretability in AI models discussed
- Attention scores can provide partial transparency by offering insights into model workings
- Experimental results and diagrams presented to support arguments
- Future directions for research proposed

## Summary

This paper discusses the role of attention mechanisms in Natural Language Processing (NLP) systems, particularly in Recurrent Neural Network (RNN) models. It addresses a recent claim that "Attention is not Explanation" and challenges the assumptions underlying this claim. The authors propose four alternative tests to determine when and whether attention can be used as an explanation: a simple uniform-weights baseline, variance calibration based on multiple random seed runs, a diagnostic framework using frozen weights from pretrained models, and an end-to-end adversarial attention training protocol. These tests allow for meaningful interpretation of attention mechanisms in RNN models. The authors provide evidence that even when reliable adversarial distributions are found, they do not perform well on a simple diagnostic test, indicating that prior work does not disprove the usefulness of attention mechanisms for explainability. The paper also discusses different notions of transparency, explainability, and interpretability in Artificial Intelligence (AI) models and argues that attention scores can provide partial transparency by offering insights into the inner workings of a model. The authors present experimental results and diagrams to support their arguments and propose future directions for research in this area.