# Intelligence at the Edge of Chaos

**Authors:** Shiyang Zhang, Aakash Patel, Syed A Rizvi, Nianchen Liu, Sizhuang He, Amin Karbasi, Emanuele Zappala, David van Dijk  
**arXiv:** [2410.02536](https://arxiv.org/abs/2410.02536)  
**Date:** 2024-10-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2410.02536v1/

## Key points

- Study focuses on emergence of intelligent behavior in artificial systems by examining impact of rule complexity on models trained to predict rules
- Investigates relationship between rule complexity and intelligence displayed by Large Language Models (LLMs) through performance on downstream tasks
- Higher complexity leads to greater intelligence in models, as seen in proficiency in reasoning and chess move prediction tasks
- Intelligence stems from ability to predict complexity; exposure to complexity may be crucial for development
- Models trained on more complex rules heavily rely on past states for prediction, with attention patterns showing stronger focus on recent states compared to uniform or highly chaotic rules
- Models trained on complex rules excel at predicting immediate next states and capture complex patterns beyond traditional rule-based approaches
- Importance of system complexity in shaping emergent intelligence in artificial systems emphasized
- Pretraining models on more complex rules within ECAs improves downstream performance up to a certain level before diminishing returns set in

## Summary

In this study, we explore the emergence of intelligent behavior in artificial systems by examining the impact of rule complexity on the capabilities of models trained to predict these rules. Our focus is on elementary cellular automata (ECA), simple yet powerful one-dimensional systems that exhibit behaviors ranging from trivial to highly complex. By training different Large Language Models (LLMs) on various ECAs, we investigate the relationship between rule complexity and intelligence displayed by the models through their performance on downstream tasks. Our findings suggest that higher complexity leads to greater intelligence in models, as seen in their proficiency in reasoning and chess move prediction tasks. This highlights a "sweet spot" of complexity that fosters intelligence. We propose that intelligence stems from the ability to predict complexity and suggest that exposure to complexity may be crucial for its development. Furthermore, our analysis reveals that models trained on more complex rules heavily rely on past states for prediction, with attention patterns showing a stronger focus on recent states compared to uniform or highly chaotic rules. Additionally, our comparison between short-term and long-term prediction tasks demonstrates that models trained on complex rules not only excel at predicting immediate next states but also capture complex patterns beyond traditional rule-based approaches. Overall, our study emphasizes the importance of system complexity in shaping emergent intelligence in artificial systems. By pretraining models on more complex rules within ECAs, we observe improved downstream performance up to a certain level before diminishing returns set in. This research sheds light on the intricate interplay between system complexity and intelligent behavior and offers insights into optimizing model training for enhanced predictive capabilities.