# Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Models

**Authors:** Marianna Nezhurina, Lucia Cipolina-Kun, Mehdi Cherti, Jenia Jitsev  
**arXiv:** [2406.02061](https://arxiv.org/abs/2406.02061)  
**Date:** 2024-06-04  
**Summary page:** https://www.summarizepaper.com/en/paper/2406.02061v1/

## Key points

- Large Language Models (LLMs) have demonstrated strong performance in various tasks and conditions in a few-shot or zero-shot manner.
- Scaling laws suggest that LLMs show improved functionality with increased pre-training scale.
- A recent study revealed significant limitations in the reasoning capabilities of state-of-the-art LLMs, particularly in common sense problems easily solvable by humans.
- LLMs trained at large scales exhibited overconfidence in incorrect solutions and provided nonsensical explanations for their failures.
- Standard interventions like enhanced prompting or multi-step re-evaluation did not correct the erroneous responses, raising concerns about the true capabilities of current generation LLMs.

## Summary

, , , , 

In recent advancements in the field of transferable learning, Large Language Models (LLMs) have emerged as key players, demonstrating strong performance across various tasks and conditions in a few-shot or zero-shot manner. These models, often referred to as foundation models, exhibit scaling laws that suggest improved functionality with increased pre-training scale. However, a recent study has uncovered significant limitations in the reasoning capabilities of state-of-the-art LLMs when faced with simple common sense problems that are easily solvable by humans. The study revealed a striking breakdown in function and reasoning abilities of LLMs trained at large scales, despite their claims of strong performance. Notably, these models displayed overconfidence in incorrect solutions and provided nonsensical explanations to justify their failures. Standard interventions such as enhanced prompting or multi-step re-evaluation failed to correct the erroneous responses. These findings raise concerns about the true capabilities of current generation LLMs and highlight deficiencies in existing language model benchmarks, particularly in assessing reasoning abilities. While these models may excel in complex real-world tasks like graduate exams, they struggle with basic common sense reasoning tasks. As a result, there is a call to action for the scientific community to re-assess the reasoning capabilities of LLMs by developing standardized benchmarks specifically designed to detect such deficits. This collaborative effort aims to pave the way for improving the current shortcomings in evaluating language models and ensuring more accurate assessments of their capabilities. Furthermore, additional research is needed to address these challenges and enhance our understanding of LLMs' reasoning abilities. The study provides valuable insights into the limitations of current evaluation procedures and emphasizes the importance of creating robust benchmarks for future advancements in natural language processing technology.