# Looped Transformers as Programmable Computers

**Authors:** Angeliki Giannou, Shashank Rajput, Jy-yong Sohn, Kangwook Lee, Jason D. Lee, Dimitris Papailiopoulos  
**arXiv:** [2301.13196](https://arxiv.org/abs/2301.13196)  
**Date:** 2023-01-30  
**Summary page:** https://www.summarizepaper.com/en/paper/2301.13196v1/

## Key points

- The paper is titled "Looped Transformers as Programmable Computers"
- The authors present a framework for using transformer networks as universal computers
- Transformers are programmed with specific weights and placed in a loop, with the input sequence acting as a punchcard containing instructions and memory for data read/writes
- A constant number of encoder layers can emulate basic computing blocks such as embedding edit operations, non-linear functions, function calls, program counters, and conditional branches
- Using these building blocks, they are able to emulate a small instruction-set computer
- This allows them to map iterative algorithms to programs that can be executed by a looped 13-layer transformer
- The transformer can be instructed by its input to emulate various tasks such as a basic calculator, a basic linear algebra library, and in-context learning algorithms that employ backpropagation
- The work highlights the versatility of the attention mechanism and demonstrates its capability to execute full-fledged general-purpose programs even with shallow transformers.
- This paper presents an innovative approach for utilizing transformer networks as programmable computers which opens up new avenues for research in the fields of machine learning and artificial intelligence.

## Summary

In their paper titled "Looped Transformers as Programmable Computers," Angeliki Giannou, Shashank Rajput, Jy-yong Sohn, Kangwook Lee, Jason D. Lee, and Dimitris Papailiopoulos present a framework for using transformer networks as universal computers. They achieve this by programming the transformers with specific weights and placing them in a loop. The input sequence acts as a punchcard that contains instructions and memory for data read/writes. The authors demonstrate that a constant number of encoder layers can emulate basic computing blocks such as embedding edit operations, non-linear functions, function calls, program counters, and conditional branches. By using these building blocks, they are able to emulate a small instruction-set computer. This allows them to map iterative algorithms to programs that can be executed by a looped 13-layer transformer. The authors show how this transformer can be instructed by its input to emulate various tasks such as a basic calculator, a basic linear algebra library, and in-context learning algorithms that employ backpropagation. Their work highlights the versatility of the attention mechanism and demonstrates its capability to execute full-fledged general-purpose programs even with shallow transformers. Overall, this paper presents an innovative approach for utilizing transformer networks as programmable computers which opens up new avenues for research in the fields of machine learning and artificial intelligence.