# Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach

**Authors:** Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, Tom Goldstein  
**arXiv:** [2502.05171](https://arxiv.org/abs/2502.05171)  
**Date:** 2025-02-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2502.05171v1/

## Key points

- Introduction of a novel language model architecture for improved test-time computation through latent reasoning
- Utilization of a recurrent block for achieving arbitrary depth at test-time instead of generating more tokens
- Ability to work effectively with small context windows without relying on specialized training data
- Capability to capture nuanced forms of reasoning that are challenging to express in words
- Scaling up the proof-of-concept model to 3.5 billion parameters and 800 billion tokens
- Demonstrated significant performance improvements on reasoning benchmarks, equivalent to a computation load of up to 50 billion parameters
- Potential of the model in enhancing reasoning capabilities without requiring extensive computational resources

## Summary

In their study, Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele and Tom Goldstein introduce a novel language model architecture designed to improve test-time computation through latent reasoning. This innovative approach diverges from traditional models by utilizing a recurrent block to achieve arbitrary depth at test-time instead of generating more tokens. The model does not rely on specialized training data and can effectively work with small context windows. Additionally, it has the capability to capture nuanced forms of reasoning that are challenging to express in words. The researchers successfully scaled up a proof-of-concept model to an impressive 3.5 billion parameters and 800 billion tokens. Through rigorous testing on reasoning benchmarks, they demonstrated significant performance improvements – sometimes even dramatically so – equivalent to a computation load of up to 50 billion parameters. This advancement showcases the potential of their model in enhancing reasoning capabilities without the need for extensive computational resources.