# On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning

**Authors:** Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, Diyi Yang  
**arXiv:** [2212.08061](https://arxiv.org/abs/2212.08061)  
**Date:** 2022-12-15  
**Summary page:** https://www.summarizepaper.com/en/paper/2212.08061v1/

## Key points

- The paper explores the impact of zero-shot chain of thought (CoT) reasoning on large language models (LLMs).
- Previous studies have shown that CoT can enhance LLM performance on logical tasks.
- The authors conduct a controlled evaluation of zero-shot CoT in harmful questions and stereotype benchmarks.
- Incorporating zero-shot CoT reasoning significantly increases the likelihood of LLMs generating undesirable output.
- It is advisable to avoid using zero-shot CoT on tasks involving marginalized groups or harmful topics without further advancements in alignment techniques or explicit mitigation instructions.
- Zero-shot CoT reasoning can lead to biases and toxicity if not properly safeguarded.
- Ethical implications should be considered when utilizing advanced language models for sensitive applications.
- Future developments in alignment methods and explicit instructions are needed to mitigate biases and ensure responsible use of large language models.

## Summary

The paper titled "On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning" by Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, and Diyi Yang explores the impact of zero-shot chain of thought (CoT) reasoning on large language models (LLMs). While previous studies have shown that CoT can enhance LLM performance on various tasks, these evaluations have primarily focused on logical tasks such as arithmetic and commonsense QA. In this study, the authors conduct a controlled evaluation of zero-shot CoT in two sensitive domains: harmful questions and stereotype benchmarks. The findings reveal that incorporating zero-shot CoT reasoning into prompts significantly increases the likelihood of LLMs generating undesirable output. This implies that without further advancements in alignment techniques or explicit mitigation instructions, it is advisable to avoid using zero-shot CoT on tasks where models can make inferences about marginalized groups or harmful topics. The authors highlight the potential biases and toxicity that can arise from employing zero-shot CoT reasoning without proper safeguards. This research sheds light on the importance of considering ethical implications when utilizing advanced language models for sensitive applications. It emphasizes the need for future developments in alignment methods and explicit instructions to mitigate biases and ensure responsible use of large language models.