# Transformers Can Do Arithmetic with the Right Embeddings

**Authors:** Sean McLeish, Arpit Bansal, Alex Stein, Neel Jain, John Kirchenbauer, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, Jonas Geiping, Avi Schwarzschild, Tom Goldstein  
**arXiv:** [2405.17399](https://arxiv.org/abs/2405.17399)  
**Date:** 2024-05-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2405.17399v1/

## Key points

- Authors address poor performance of transformers on arithmetic tasks due to inability to track position of digits within large number span
- Proposed enhancement improves performance on arithmetic tasks and multi-step reasoning tasks like sorting and multiplication
- Enhancement allows for architectural modifications such as input injection and recurrent layers to further improve accuracy

## Summary

In their paper titled "Transformers Can Do Arithmetic with the Right Embeddings," authors Sean McLeish, Arpit Bansal, Alex Stein, Neel Jain, John Kirchenbauer, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, Jonas Geiping, Avi Schwarzschild, and Tom Goldstein address the issue of poor performance of transformers on arithmetic tasks due to their inability to accurately track the position of digits within a large number span. To overcome this limitation and improve performance on arithmetic tasks and other multi-step reasoning tasks like sorting and multiplication  This enhancement not only boosts performance but also allows for architectural modifications such as input injection and recurrent layers to further improve accuracy.