# Unleashing the Creative Mind: Language Model As Hierarchical Policy For Improved Exploration on Challenging Problem Solving

**Authors:** Zhan Ling, Yunhao Fang, Xuanlin Li, Tongzhou Mu, Mingu Lee, Reza Pourreza, Roland Memisevic, Hao Su  
**arXiv:** [2311.00694](https://arxiv.org/abs/2311.00694)  
**Date:** 2023-11-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2311.00694v1/

## Key points

- Large language models (LLMs) struggle with complex reasoning tasks in mathematics
- Current approaches involve sampling or searching detailed reasoning chains but have limitations
- Proposed approach leverages LLMs as hierarchical policies through in-context learning
- Approach consists of a visionary leader and a follower for problem-solving tactics
- Follower explores multiple reasoning chains guided by leader's hints
- Tournament-based approach used to select the best solution group
- Enhances problem-solving strategy exploration and improves accuracy on challenging problems
- Code for the approach will be released at a specific GitHub repository
- Unleashes LLMs' creative potential and improves reasoning abilities in challenging tasks across domains

## Summary

Large language models (LLMs) have shown great potential in various domains but still struggle with complex reasoning tasks such as mathematical proofs and advanced mathematics problems. Current approaches to address this challenge involve sampling or searching detailed and low-level reasoning chains but have limitations in exploring the vast solution space, making it difficult for correct solutions to stand out. To overcome these limitations, we propose a novel approach that leverages the creative potential of LLMs by framing them as hierarchical policies through in-context learning. Our approach consists of a visionary leader and a follower. The leader generates multiple diverse high-level problem-solving tactics as hints while the follower executes detailed problem-solving processes based on each hint provided by the leader. The follower explores multiple reasoning chains guided by the leader's directives and generates a solution group for each proposed tactic. In addition to this hierarchical policy framework, we introduce an effective tournament-based approach to select the best solution group among those explored by the follower. This approach enhances problem-solving strategy exploration and improves accuracy on challenging problems in the MATH dataset. Our work contributes meaningful and inspiring hints for problem solving, expands exploration capabilities of LLMs, and ultimately improves the accuracy of final answers. The code for our approach will be released at https://github.com/lz1oceani/LLM-As-Hierarchical-Policy. Overall, our proposed method unleashes LLMs' creative potential by enabling them to explore multiple diverse problem-solving strategies through a hierarchical policy framework. This advancement has significant implications for improving reasoning abilities in challenging tasks across various domains.