# Language Models Represent Space and Time

**Authors:** Wes Gurnee, Max Tegmark  
**arXiv:** [2310.02207](https://arxiv.org/abs/2310.02207)  
**Date:** 2023-10-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2310.02207v1/

## Key points

- Large language models (LLMs) have capabilities that have been debated
- LLMs may develop a coherent understanding of the underlying data generating process
- The study analyzes the learned representations of Llama-2 family models across various spatial and temporal datasets
- Spatial analysis includes examining world places, US places, and NYC places datasets
- LLMs learn linear representations of space at multiple scales, consistent with variations in prompts
- Representations are unified across different types of entities such as cities and landmarks
- "Space neurons" consistently encode spatial coordinates
- Temporal analysis includes historical figures, artworks, and news headlines datasets
- LLMs also learn linear representations of time, consistent with prompting variations and entity types
- Findings suggest that modern LLMs acquire structured knowledge about space and time beyond superficial statistics
- Individual "time neurons" and "space neurons" reliably encode temporal and spatial coordinates within LLMs
- Base Llama-2 series of auto-regressive transformer language models with varying parameter sizes are used for analysis
- Linear ridge regression probes are employed to predict target labels associated with time or latitude/longitude coordinates based on network activations
- High predictive performance indicates presence of temporal and spatial information in LLM representations
- Work builds upon prior research on factual recall in LLMs and interpretability literature
- Concludes that modern LLMs develop structured knowledge about space and time beyond superficial statistics

## Summary

The capabilities of large language models (LLMs) have been a topic of debate, with some questioning whether these models simply learn superficial statistics or if they develop a coherent understanding of the underlying data generating process. In this study, we aim to provide evidence for the latter by analyzing the learned representations of Llama-2 family models across various spatial and temporal datasets. For our spatial analysis, we examine three datasets: world places, US places, and NYC places. We find that LLMs learn linear representations of space at multiple scales, which remain consistent even with variations in prompts. These representations are also unified across different types of entities such as cities and landmarks. Additionally, we identify specific "space neurons" that consistently encode spatial coordinates. In our temporal analysis, we investigate three datasets: historical figures, artworks, and news headlines. By probing the LLMs' activations on these datasets, we discover that they also learn linear representations of time. Similar to the spatial analysis, these representations are robust to prompting variations and demonstrate consistency across different types of entities. Our findings suggest that modern LLMs acquire structured knowledge about fundamental dimensions like space and time. This supports the notion that these models go beyond learning superficial statistics and instead develop literal world models. Furthermore, our study highlights the presence of individual "time neurons" and "space neurons" within LLMs that reliably encode temporal and spatial coordinates. To conduct our analysis, we utilize the base Llama-2 series of auto-regressive transformer language models with varying parameter sizes. We preprocess each dataset by running entity names through the model and saving the activations of the hidden state on the last entity token for each layer. We employ linear ridge regression probes to predict target labels associated with either time or two-dimensional latitude and longitude coordinates based on the network activations. High predictive performance indicates that LLMs possess temporal and spatial information in their representations. Our work builds upon prior research on factual recall in LLMs and draws from the interpretability literature. We contribute to the understanding of continuous facts and highlight the linear structure present in LLMs' representations. In conclusion, our analysis provides evidence that modern LLMs develop structured knowledge about space and time. These models go beyond superficial statistics and acquire literal world models.