# Unified View of Grokking, Double Descent and Emergent Abilities: A Perspective from Circuits Competition

**Authors:** Yufei Huang, Shengding Hu, Xu Han, Zhiyuan Liu, Maosong Sun  
**arXiv:** [2402.15175](https://arxiv.org/abs/2402.15175)  
**Date:** 2024-02-23  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.15175v1/

## Key points

- Recent studies in deep learning have revealed intriguing phenomena such as grokking, double descent, and emergent abilities in large language models.
- A comprehensive framework has been developed to provide a unified view of these three phenomena, focusing on the competition between memorization and generalization circuits within neural models.
- The framework delineates four distinct training dynamics based on varying combinations of model size and training data quantity.
- Detailed analysis of the double descent phenomenon has led to two verifiable predictions regarding its occurrence, supported by experimental results.
- The framework has been extended to incorporate multi-task learning paradigms, showing how algorithm tasks can lead to emergent abilities in large language models.

## Summary

, , , , 

Recent studies in deep learning have revealed intriguing phenomena such as grokking, double descent, and emergent abilities in large language models. These phenomena challenge human intuition and are essential for a deeper understanding of neural models. In response to these findings, a comprehensive framework has been developed to provide a unified view of these three phenomena, with a focus on the competition between memorization and generalization circuits within neural models. Initially used to explain grokking, this framework has been expanded to encompass a wider range of model sizes and training data volumes. It delineates four distinct training dynamics that depend on varying combinations of model size and training data quantity. Through the utilization of this framework, a detailed analysis of the double descent phenomenon has been conducted, leading to the proposal of two verifiable predictions regarding its occurrence, both supported by experimental results. Furthermore, the framework has been extended to incorporate multi-task learning paradigms, demonstrating how algorithm tasks can give rise to emergent abilities in large language models. This novel perspective sheds light on the understanding of emergent abilities within neural models. In conclusion, this research introduces an innovative framework designed to analyze different training dynamics across various model sizes and training dataset quantities by examining the competition between memorization and generalization circuits. By exploring double descent and inducing emergent behavior in generalization tasks through this framework, new insights into deep learning mechanisms have been gained. While there is still room for further exploration beyond algorithm tasks in future research efforts, this work contributes significantly to advancing our mechanistic understanding of large language models.