# Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinking

**Authors:** Yilong Chen, Junyuan Shang, Zhenyu Zhang, Yanxi Xie, Jiawei Sheng, Tingwen Liu, Shuohuan Wang, Yu Sun, Hua Wu, Haifeng Wang  
**arXiv:** [2502.13842](https://arxiv.org/abs/2502.13842)  
**Date:** 2025-02-19  
**Summary page:** https://www.summarizepaper.com/en/paper/2502.13842v1/

## Key points

- Large language models (LLMs) face challenges in achieving optimal performance within model parameter constraints
- Critical tokens requiring intricate reasoning abilities can lead to sudden spikes in gradients across layers, highlighting stress points in standard Transformers' architecture
- Inner Thinking Transformer (ITT) introduces a novel approach by reimagining computation as implicit thinking steps for more efficient resource allocation
- ITT features Adaptive Token Routing for dynamic computation assignment, Residual Thinking Connections for iterative refinement, and Thinking Step Encoding for reasoning phase differentiation
- ITT enables deeper processing of critical tokens without expanding model parameters, achieving up to 96.5% performance compared to larger Transformers with fewer parameters
- ITT reduces training data by 43.2% and outperforms Transformer/Loop variants in 11 benchmark tests
- Elastic computation allocation during inference is possible with ITT, optimizing implicit thinking pathways for improved performance and efficiency

## Summary

In the realm of large language models (LLMs), there exists a significant challenge in achieving optimal performance within the constraints of model parameters. This is particularly evident when processing critical tokens that require intricate reasoning abilities. Through empirical analysis, it has been observed that these challenging tokens can lead to sudden spikes in gradients across various layers, thereby highlighting stress points within standard Transformers' architecture. To address this issue, a novel approach known as the Inner Thinking Transformer (ITT) has been introduced. ITT reimagines the computation process within layers as implicit thinking steps, allowing for more efficient allocation of resources. One key feature of ITT is Adaptive Token Routing, which dynamically assigns computation based on the specific requirements of each token. Additionally, Residual Thinking Connections are utilized to iteratively refine representations, while Thinking Step Encoding helps differentiate between different phases of reasoning. By implementing ITT, deeper processing of critical tokens becomes possible without the need for expanding model parameters. Evaluations conducted across models ranging from 162M to 466M parameters have shown that ITT can achieve up to 96.5% performance compared to a 466M Transformer using only 162M parameters. Furthermore, ITT reduces training data by 43.2% and surpasses Transformer/Loop variants in 11 benchmark tests. One notable advantage of ITT is its ability to enable elastic computation allocation during inference, striking a balance between performance and efficiency through optimized implicit thinking pathways within the architecture. Overall, ITT represents a promising advancement in enhancing the capabilities of large language models while overcoming inherent performance bottlenecks under parameter constraints.