# Formal Algorithms for Transformers

**Authors:** Mary Phuong, Marcus Hutter  
**arXiv:** [2207.09238](https://arxiv.org/abs/2207.09238)  
**Date:** 2022-07-19  
**Summary page:** https://www.summarizepaper.com/fr/paper/2207.09238v1/

## Abstract

This document aims to be a self-contained, mathematically precise overview of transformer architectures and algorithms (*not* results). It covers what transformers are, how they are trained, what they are used for, their key architectural components, and a preview of the most prominent models. The reader is assumed to be familiar with basic ML terminology and simpler neural network architectures such as MLPs.