# Attention Is All You Need

**Authors:** Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin  
**arXiv:** [1706.03762](https://arxiv.org/abs/1706.03762)  
**Date:** 2017-06-12  
**Summary page:** https://www.summarizepaper.com/en/paper/1706.03762v6/

## Key points

- Authors propose a new network architecture called the Transformer
- Transformer is based solely on attention mechanisms, eliminating the need for recurrent or convolutional neural networks in an encoder-decoder configuration
- Transformer models outperformed existing models in machine translation tasks
- Achieved a BLEU score of 28.4 on WMT 2014 English-to-German translation task, surpassing previous best results by over 2 BLEU points
- Achieved a state-of-the-art BLEU score of 41.8 on WMT 2014 English-to-French translation task after only 3.5 days of training on eight GPUs
- Transformer model generalizes well to other tasks, such as English constituency parsing with large and limited training data sets
- Offers advantages such as improved quality, increased parallelizability, and reduced training time compared to other models used for similar tasks

## Summary

In the paper titled "Attention Is All You Need," authors Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin propose a new network architecture called the Transformer. This architecture is based solely on attention mechanisms and eliminates the need for recurrent or convolutional neural networks in an encoder-decoder configuration. The authors conducted experiments on two machine translation tasks and found that the Transformer models outperformed existing models in terms of quality while also being more parallelizable and requiring less training time. Specifically, their model achieved a BLEU score of 28.4 on the WMT 2014 English-to-German translation task which surpassed previous best results by over 2 BLEU points. On the WMT 2014 English-to-French translation task their model achieved a state-of-the-art BLEU score of 41.8 after only 3.5 days of training on eight GPUs. Furthermore, they demonstrated that the Transformer generalizes well to other tasks by successfully applying it to English constituency parsing with both large and limited training data sets. Overall, this paper presents a novel network architecture that relies solely on attention mechanisms and achieves superior performance in machine translation tasks compared to existing models. The Transformer model offers advantages such as improved quality, increased parallelizability and reduced training time when compared to other models used for similar tasks.