# When do you need Chain-of-Thought Prompting for ChatGPT?

**Authors:** Jiuhai Chen, Lichang Chen, Heng Huang, Tianyi Zhou  
**arXiv:** [2304.03262](https://arxiv.org/abs/2304.03262)  
**Date:** 2023-04-06  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.03262v1/

## Key points

- Chain-of-Thought (CoT) prompting is effective in eliciting complex multi-step reasoning from Large Language Models (LLMs)
- Adding the CoT instruction "Let's think step-by-step" improved GPT-3's accuracy from 17.7% to 78.7%
- CoT is no longer effective for certain tasks like arithmetic reasoning on ChatGPT, but still effective on other reasoning tasks
- ChatGPT may have already been trained on these tasks with CoT and thus memorized the instruction, highlighting a potential risk of overfitting/bias toward instructions introduced in IFT
- ChatGPT demonstrates strong reasoning capability without the guidance of CoT prompting for arithmetic reasoning tasks, suggesting that some arithmetic datasets may be included in the pre-training mix
- The study sheds light on the importance of understanding LLMs' behavior in reasoning tasks and highlights potential risks associated with IFT and pretraining dataset leakage

## Summary

The use of Chain-of-Thought (CoT) prompting has been shown to effectively elicit complex multi-step reasoning from Large Language Models (LLMs). For instance, adding the CoT instruction "Let's think step-by-step" to each input query of the MultiArith dataset improved GPT-3's accuracy from 17.7% to 78.7%. However, it is unclear whether CoT is still effective on more recent instruction finetuned (IFT) LLMs such as ChatGPT. Surprisingly, on ChatGPT, CoT is no longer effective for certain tasks like arithmetic reasoning while still being effective on other reasoning tasks. Moreover, ChatGPT usually achieves the best performance on these former tasks and can generate CoT even without being instructed to do so. This suggests that ChatGPT may have already been trained on these tasks with CoT and thus memorized the instruction. The authors' analysis highlights a potential risk of overfitting/bias toward instructions introduced in IFT, which is becoming more common in training LLMs. Additionally, it indicates possible leakage of the pretraining recipe where one can verify whether a dataset and instruction were used in training ChatGPT. The experiments conducted report new baseline results of ChatGPT on various reasoning tasks and provide novel insights into LLM's profiling, instruction memorization, and pretraining dataset leakage. Further investigation reveals that ChatGPT demonstrates strong reasoning capability without the guidance of CoT prompting for arithmetic reasoning tasks. In contrast to these arithmetic tasks, CoT prompting is still required to elicit the reasoning ability in ChatGPT for other reasoning tasks. These observations suggest that some arithmetic datasets may be included in the pre-training mix, leading to ChatGPT memorizing instructions and autonomously generating rationales. The authors' analyses underscore several fundamental challenges that need addressing in future research such as how the pre-training recipe affects the performance of the inference stage and how to effectively identify whether certain datasets were employed for pre-training or detect data leakage issues. Overall, this study sheds light on the importance of understanding LLMs' behavior in reasoning tasks and highlights potential risks associated with IFT and pretraining dataset leakage.