# Let's Think Dot by Dot: Hidden Computation in Transformer Language Models

**Authors:** Jacob Pfau, William Merrill, Samuel R. Bowman  
**arXiv:** [2404.15758](https://arxiv.org/abs/2404.15758)  
**Date:** 2024-04-24  
**Summary page:** https://www.summarizepaper.com/en/paper/2404.15758v1/

## Key points

- Researchers explore the use of filler tokens in language models to improve performance on algorithmic tasks
- Transformers can effectively utilize meaningless filler tokens, such as '......,' to solve challenging algorithmic tasks
- Learning to use filler tokens effectively requires specific and dense supervision for convergence
- Theoretical framework provided for identifying problems where filler tokens are beneficial based on quantifier depth of a first-order formula
- Additional tokens can offer computational advantages independent of token choice
- Concerns raised about large language models engaging in unauditable hidden computations detached from observable chain-of-thought tokens when intermediate tokens serve as fillers
- Filler tokens can match the performance of chain-of-thought reasoning on certain tasks

## Summary

In their paper titled "Let's Think Dot by Dot: Hidden Computation in Transformer Language Models," Jacob Pfau, William Merrill, and Samuel R. Bowman explore the use of filler tokens in language models to improve performance on algorithmic tasks. The authors investigate whether the observed performance gains in language models can be attributed to human-like task decomposition or simply the increased computational capacity provided by additional tokens. The study demonstrates that transformers can effectively utilize meaningless filler tokens, such as '......,' to solve challenging algorithmic tasks that they previously struggled with when responding without intermediate tokens. However, the researchers note that learning to use filler tokens effectively requires specific and dense supervision for convergence. Furthermore, the authors provide a theoretical framework for identifying problems where filler tokens are beneficial based on the quantifier depth of a first-order formula. They argue that for problems meeting this characterization, chain-of-thought tokens may not necessarily provide information about the intermediate computational steps involved in multi-token computations. Overall, the results suggest that additional tokens can offer computational advantages independent of token choice. However, concerns are raised about large language models engaging in unauditable hidden computations detached from observable chain-of-thought tokens when intermediate tokens serve as fillers. The study also includes a detailed analysis and comparison of different LM question-answering protocols, including chain of thought reasoning, filler token usage, and immediate answer approaches. Through experiments, it is shown that filler tokens can match the performance of chain-of-thought reasoning on certain tasks. Additionally, the authors make their code available for further exploration and replication of their findings at https://github.com/JacobPfau/fillerTokens. This research sheds light on how transformer language models leverage filler tokens for improved computational performance and raises important considerations regarding hidden computations within these models.