# RoFormer: Enhanced Transformer with Rotary Position Embedding

**Authors:** Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, Yunfeng Liu  
**arXiv:** [2104.09864](https://arxiv.org/abs/2104.09864)  
**Date:** 2021-04-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2104.09864v5/

## Key points

- The paper explores the effectiveness of position encoding in the transformer architecture
- The authors propose a novel approach called RoPE to leverage positional information effectively
- RoPE encodes absolute position using a rotation matrix and incorporates explicit relative position dependency in self-attention formulation
- Advantages of RoPE include flexibility in sequence length, decaying inter-token dependency with increasing relative distances, and the ability to equip linear self-attention with relative position encoding
- Experimental results consistently demonstrate that RoFormer outperforms alternative methods on various long text classification benchmark datasets
- The paper provides a theoretical analysis to explain some of the experimental findings
- RoFormer has already been integrated into Huggingface, a popular natural language processing library.

## Summary

The paper titled "RoFormer: Enhanced Transformer with Rotary Position Embedding" explores the effectiveness of position encoding in the transformer architecture. allows for valuable supervision in modeling dependencies between elements at different positions within a sequence. The authors investigate various methods to integrate positional information into transformer-based language models and propose a novel approach called  to leverage this information effectively. RoPE encodes absolute position using a rotation matrix and incorporates explicit relative position dependency in self-attention formulation. This approach offers several advantages, including flexibility in sequence length, decaying inter-token dependency with increasing relative distances, and the ability to equip linear self-attention with relative position encoding. To evaluate the enhanced transformer with rotary position embedding, also known as , the authors conduct experiments on various long text classification benchmark datasets. The results consistently demonstrate that RoFormer outperforms alternative methods. Additionally, the paper provides a theoretical analysis to explain some of the experimental findings. It is worth noting that RoFormer has already been integrated into Huggingface, a popular natural language processing library. Further details about RoFormer can be found in the Huggingface documentation. Authors: Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, Yunfeng Liu. Title: RoFormer: Enhanced Transformer with Rotary Position Embedding. Abstract: The paper investigates the effectiveness of position encoding in transformers and proposes a novel method called . RoPE encodes absolute position using a rotation matrix and incorporates explicit relative position dependency in self-attention formulation. The authors evaluate  on various long text classification benchmark datasets and show consistent improvements over alternative methods. The paper also provides a theoretical analysis to explain experimental results.