# Explaining grokking through circuit efficiency

**Authors:** Vikrant Varma, Rohin Shah, Zachary Kenton, János Kramár, Ramana Kumar  
**arXiv:** [2309.02390](https://arxiv.org/abs/2309.02390)  
**Date:** 2023-09-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.02390v1/

## Key points

- Grokking refers to the phenomenon where a network with perfect training accuracy but poor generalization can transition to perfect generalization with further training.
- The authors propose that grokking occurs when a task has both a generalizing solution and a memorizing solution.
- Generalizing solutions are slower to learn but more efficient, while memorizing solutions are faster to learn but less efficient.
- As training datasets become larger, memorizing circuits become increasingly inefficient while generalizing circuits remain relatively unaffected.
- There exists a critical dataset size at which memorization and generalization are equally efficient.
- The authors make and confirm four novel predictions about grokking, providing strong evidence for their hypothesis.
- Two unexpected behaviors related to grokking are ungrokking (regression from perfect to low test accuracy after further training) and semi-grokking (delayed generalization to partial rather than perfect test accuracy).
- This study sheds light on the puzzling phenomenon of grokking in neural networks by proposing an explanation based on circuit efficiency.
- The findings contribute to our understanding of neural network generalization and have implications for improving model performance in various tasks.

## Summary

In their paper titled "Explaining grokking through circuit efficiency," authors Vikrant Varma, Rohin Shah, Zachary Kenton, János Kramár, and Ramana Kumar explore the phenomenon of grokking in neural network generalization. Grokking refers to the surprising observation that a network with perfect training accuracy but poor generalization can transition to perfect generalization with further training. The authors propose that grokking occurs when a task has both a generalizing solution and a memorizing solution. The generalizing solution is slower to learn but more efficient, producing larger logits with the same parameter norm. On the other hand, the memorizing solution is faster to learn but less efficient. They hypothesize that as training datasets become larger, memorizing circuits become increasingly inefficient while generalizing circuits remain relatively unaffected. This suggests that there exists a critical dataset size at which memorization and generalization are equally efficient. To support their explanation, the authors make and confirm four novel predictions about grokking. These predictions provide strong evidence in favor of their hypothesis and demonstrate two unexpected behaviors: ungrokking and semi-grokking. Ungrokking refers to the regression of a network from perfect to low test accuracy after further training whereas semi-grokking occurs when a network shows delayed generalization to partial rather than perfect test accuracy. Overall, this study sheds light on the puzzling phenomenon of grokking in neural networks by proposing an explanation based on circuit efficiency. The findings contribute to our understanding of neural network generalization and have implications for improving model performance in various tasks.