# Evaluating Cognitive Maps and Planning in Large Language Models with CogEval

**Authors:** Ida Momennejad, Hosein Hasanbeig, Felipe Vieira, Hiteshi Sharma, Robert Osazuwa Ness, Nebojsa Jojic, Hamid Palangi, Jonathan Larson  
**arXiv:** [2309.15129](https://arxiv.org/abs/2309.15129)  
**Date:** 2023-09-25  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.15129v1/

## Key points

- Growing interest in exploring cognitive abilities of large language models (LLMs)
- Proposal of CogEval - a cognitive science-inspired protocol for systematic evaluation of cognitive capacities in LLMs
- Application of CogEval to evaluate cognitive maps and planning ability across eight different LLMs
- Task prompts based on human experiments with established construct validity
- Findings reveal significant failure modes in planning tasks, such as hallucinations and getting trapped in loops
- Lack of emergent out-of-the-box planning ability in LLMs
- Possible explanation: LLMs lack understanding of latent relational structures underlying planning problems (cognitive maps)
- Importance of rigorous evaluation protocols like CogEval to assess true cognitive capabilities of LLMs
- Insights into limitations of LLMs' planning abilities discussed by authors

## Summary

In recent studies, there has been a growing interest in exploring the cognitive abilities of large language models (LLMs). To address the limitations of existing studies which relied on anecdotal evidence or lacked systematic evaluation methods, the authors propose CogEval - a cognitive science-inspired protocol for the systematic evaluation of cognitive capacities in LLMs. This study applies CogEval to systematically evaluate cognitive maps and planning ability across eight different LLMs. The task prompts for evaluation are based on human experiments that offer established construct validity for evaluating planning and are not present in the LLM training sets. The findings reveal that while LLMs demonstrate apparent competence in some planning tasks with simpler structures, systematic evaluation uncovers significant failure modes in planning tasks such as hallucinations of invalid trajectories and getting trapped in loops. Consequently, these findings do not support the notion of emergent out-of-the-box planning ability in LLMs. One possible explanation is that LLMs lack an understanding of the latent relational structures underlying planning problems known as cognitive maps. The implications of these findings for application and future directions are discussed by the authors. Overall, this study highlights the importance of rigorous evaluation protocols like CogEval to assess the true cognitive capabilities of large language models and provides insights into their limitations when it comes to planning abilities.