# Self-Attention with Relative Position Representations

**Authors:** Peter Shaw, Jakob Uszkoreit, Ashish Vaswani  
**arXiv:** [1803.02155](https://arxiv.org/abs/1803.02155)  
**Date:** 2018-03-06  
**Summary page:** https://www.summarizepaper.com/en/paper/1803.02155v1/

## Key points

- Authors: Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani
- Introduction of an alternative approach to the Transformer model for machine translation
- Focus on incorporating relative position representations in addition to absolute positions
- Significant improvements in translation quality by considering relative positions
- Outperforms using only absolute position representations on popular translation tasks
- Practical implementation as a form of relation-aware self-attention mechanism
- Contribution of valuable insights into enhancing machine translation models

## Summary

In their paper titled "Self-Attention with Relative Position Representations," authors Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani introduce an alternative approach to the Transformer model for machine translation. The Transformer was developed by Vaswani et al. in 2017 and has shown state-of-the-art performance in translation tasks by relying solely on an attention mechanism. Unlike traditional recurrent and convolutional neural networks, the Transformer does not explicitly incorporate relative or absolute position information into its structure. Instead, it requires the addition of representations of absolute positions to its inputs. In this work, the authors propose extending the self-attention mechanism to efficiently consider representations of relative positions, specifically focusing on the distances between sequence elements. By incorporating these relative position representations, significant improvements are achieved in translation quality. On popular translation tasks such as WMT 2014 English-to-German and English-to-French, the proposed approach outperforms using only absolute position representations by 1.3 BLEU and 0.3 BLEU respectively. Interestingly, the authors note that combining both relative and absolute position representations does not lead to further enhancements in translation quality. They describe a practical implementation of their method and frame it as a form of relation-aware self-attention mechanisms that can be applied to various graph-labeled inputs. Overall, this study contributes valuable insights into enhancing machine translation models by considering relative positions alongside absolute positions within the self-attention mechanism framework. The findings offer a promising direction for improving the efficiency and effectiveness of neural network architectures in natural language processing tasks like translation.