# Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

**Authors:** Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, Denny Zhou  
**arXiv:** [2201.11903](https://arxiv.org/abs/2201.11903)  
**Date:** 2022-01-28  
**Summary page:** https://www.summarizepaper.com/en/paper/2201.11903v6/

## Key points

- Recent advancements in natural language processing have been driven by the development of large language models, which have shown improved performance and sample efficiency.
- Scaling up model size alone has not been sufficient for achieving high performance on challenging tasks such as arithmetic, commonsense, and symbolic reasoning.
- The paper explores how the reasoning ability of large language models can be unlocked by a simple method called chain-of-thought prompting.
- The method is motivated by two ideas: generating natural language rationales that lead to the final answer for arithmetic reasoning and in-context few-shot learning via prompting.
- Ling et al. (2017) pioneered the idea of using natural language rationales to solve math word problems through a series of intermediate steps, while Cobbe et al. (2021) extended this work by creating a larger dataset and using it to finetune a pretrained language model rather than training a model from scratch.
- In program synthesis, Nye et al. (2021) leveraged language models to predict the final outputs of Python programs via step-by-step prediction.
- Chain-of-thought prompting significantly improves the ability of large language models to perform complex reasoning tasks such as arithmetic, commonsense, and symbolic reasoning.
- Experiments on three large language models demonstrate that providing a few chain-of-thought demonstrations as exemplars in prompting achieves state-of-the-art accuracy on math word problems benchmark GSM8K, surpassing even finetuned GPT-3 with a verifier.
- Chain-of-thought prompting is presented as a simple and broadly applicable method for enhancing reasoning in language models.
- The broadening range of reasoning tasks that language models can perform will hopefully inspire further work on language-based approaches to reasoning.

## Summary

Recent advancements in natural language processing have been driven by the development of large language models, which have shown improved performance and sample efficiency. However, scaling up model size alone has not been sufficient for achieving high performance on challenging tasks such as arithmetic, commonsense, and symbolic reasoning. This paper explores how the reasoning ability of large language models can be unlocked by a simple method called chain-of-thought prompting. The method is motivated by two ideas: generating natural language rationales that lead to the final answer for arithmetic reasoning and in-context few-shot learning via prompting. The paper draws inspiration from related work in various research areas, including using intermediate steps to solve reasoning problems and recent work on prompting. Ling et al. (2017) pioneered the idea of using natural language rationales to solve math word problems through a series of intermediate steps, while Cobbe et al. (2021) extended this work by creating a larger dataset and using it to finetune a pretrained language model rather than training a model from scratch. In program synthesis, Nye et al. (2021) leveraged language models to predict the final outputs of Python programs via step-by-step prediction. The paper shows that chain-of-thought prompting significantly improves the ability of large language models to perform complex reasoning tasks such as arithmetic, commonsense, and symbolic reasoning. Experiments on three large language models demonstrate that providing a few chain-of-thought demonstrations as exemplars in prompting achieves state-of-the-art accuracy on math word problems benchmark GSM8K, surpassing even finetuned GPT-3 with a verifier. In conclusion, this paper presents chain-of-thought prompting as a simple and broadly applicable method for enhancing reasoning in language models. The method unlocks emergent properties of model scale that allow sufficiently large language models to perform reasoning tasks that otherwise have flat scaling curves. The broadening range of reasoning tasks that language models can perform will hopefully inspire further work on language based approaches to reasoning.