# Emergent world representations: Exploring a sequence model trained on a synthetic task

**Authors:** Kenneth Li, Aspen K. Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, Martin Wattenberg  
**arXiv:** [2210.13382](https://arxiv.org/abs/2210.13382)  
**Date:** 2022-10-24  
**Summary page:** https://www.summarizepaper.com/en/paper/2210.13382v1/

## Key points

- Study investigates capabilities of language models and their reliance on surface statistics or internal representations in generating sequences
- Variant of GPT model called Othello-GPT used to predict legal moves in game of Othello
- Othello-GPT demonstrates impressive ability to generate legal moves despite lacking prior knowledge or training on strategic gameplay
- Model has developed nonlinear internal representation of board state
- Representation can be manipulated to control network's output
- "Latent saliency maps" created using this representation, providing human-readable explanations for predictions
- Findings shed light on how language models like GPT develop complex internal representations without explicit instruction
- Research provides insights into interpretability and capabilities of language models by studying world representations in controlled context like Othello

## Summary

This study investigates the capabilities of language models and their reliance on surface statistics or internal representations in generating sequences. To do so, a variant of the GPT model called Othello-GPT is utilized to predict legal moves in the game of Othello. Despite lacking prior knowledge or training on strategic gameplay and winning strategies, Othello-GPT demonstrates an impressive ability to generate legal moves. This suggests that the model has developed a nonlinear internal representation of the board state. Further investigation through interventional experiments reveals that this representation can be manipulated to control the network's output. The researchers also create "latent saliency maps" using this representation, providing human-readable explanations for predictions. These findings shed light on how language models like GPT can develop complex internal representations without explicit instruction. By studying world representations in a controlled context like Othello, this research provides insights into the interpretability and capabilities of language models.