# Chain-of-Thought Reasoning is a Policy Improvement Operator

**Authors:** Hugh Zhang, David C. Parkes  
**arXiv:** [2309.08589](https://arxiv.org/abs/2309.08589)  
**Date:** 2023-09-15  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.08589v1/

## Key points

- Large language models currently rely on being trained on large amounts of human-generated data
- They lack the ability to teach themselves new skills
- SECToR (Self-Education via Chain-of-Thought Reasoning) is introduced as a proof-of-concept demonstration
- SECToR shows that language models can learn new skills using chain-of-thought reasoning
- SECToR uses chain-of-thought reasoning to think through problems step-by-step and then fine-tunes the model to generate answers without relying on chain-of-thought reasoning
- Language models trained via SECToR autonomously learn to add up to 29 digit numbers without access to ground truth examples beyond initial supervised fine tuning phase
- Chain of thought reasoning can act as a policy improvement operator, similar to Monte Carlo Tree Search in AlphaZero
- This research opens up possibilities for language models to learn and teach themselves without human demonstrations
- The authors hope this work can lead to reduced reliance on large amounts of training data for language models.

## Summary

Large language models have amazed the world with their impressive capabilities, but they currently rely on being trained on large amounts of human-generated data and lack the ability to teach themselves new skills. In this paper, the authors introduce SECToR (Self-Education via Chain-of-Thought Reasoning), a proof-of-concept demonstration that shows language models can successfully learn new skills using chain-of-thought reasoning. Inspired by previous work in reinforcement learning and human cognition, SECToR first uses chain-of-thought reasoning to think through problems step-by-step. It then fine-tunes the model to generate the same answers without relying on chain-of-thought reasoning. Through this process, language models trained via SECToR autonomously learn to add up to 29 digit numbers without any access to ground truth examples beyond an initial supervised fine tuning phase. The central hypothesis is that chain of thought reasoning can act as a policy improvement operator, similar to how Monte Carlo Tree Search is used in AlphaZero. This research opens up possibilities for language models to learn and teach themselves without the need for human demonstrations. The authors hope that this work can lead to new directions in which language models can acquire new skills independently, reducing reliance on large amounts of training data.