# A Survey of Transformers

**Authors:** Tianyang Lin, Yuxin Wang, Xiangyang Liu, Xipeng Qiu  
**arXiv:** [2106.04554](https://arxiv.org/abs/2106.04554)  
**Date:** 2021-06-08  
**Summary page:** https://www.summarizepaper.com/en/paper/2106.04554v1/

## Key points

- Authors Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu conducted a comprehensive survey on Transformers in artificial intelligence.
- Transformers have advanced significantly in AI fields like natural language processing, computer vision, and audio processing.
- The survey introduces a novel taxonomy of Transformer variants known as X-formers and explores them from three perspectives: architectural modifications, pre-training techniques, and real-world applications.
- Despite the existence of numerous X-former variants, a systematic literature review on them is lacking.
- The authors provide valuable insights into the evolution and diversification of Transformer models through categorization and analysis.
- The survey outlines potential future research directions to guide researchers towards innovation in AI technologies.

## Summary

In their comprehensive survey titled "A Survey of Transformers," authors Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu delve into the realm of Transformers in artificial intelligence. This survey serves as a valuable resource for understanding the landscape of Transformer models and their implications across diverse domains within AI. Transformers have made significant strides in various AI fields such as natural language processing, computer vision, and audio processing. This has garnered immense interest from both academic and industry researchers. Despite the plethora of Transformer variants known as X-formers that have been proposed to date, a systematic and thorough literature review on these variants is notably absent. The survey begins by providing an overview of the vanilla Transformer model before introducing a novel taxonomy of X-formers. The authors then meticulously explore various X-formers from three distinct perspectives: architectural modifications, pre-training techniques, and real-world applications. By categorizing and analyzing these variants through different lenses, the survey offers valuable insights into the evolution and diversification of Transformer models. Furthermore, the authors outline potential directions for future research in the field of Transformers. By identifying areas that warrant further exploration and development, they aim to guide researchers towards new avenues for innovation and advancement in AI technologies. Overall,this survey provides a comprehensive understanding of Transformer models and their impact on diverse domains within artificial intelligence. It serves as a crucial reference for researchers looking to stay updated on the latest developments in this rapidly evolving field.