Grokking › Summary
Michael BrenndoerferNovember 11, 20252 min read
Language AI HandbookAll chapters Part 30 of 67Emergent Capabilities
Grokking › SummaryOne section · Sections in this chapter11 / 13 ↓
Sections in this chapter13
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
This chapter explored grokking, the phenomenon where neural networks suddenly generalize long after memorizing training data, overturning common view about when and how learning occurs.
Key characteristics of grokking:
- Dramatic delay between achieving perfect training accuracy and the onset of generalization
- Sharp phase transition from memorization to generalization, concentrated in a narrow window of training steps
- Critical dependence on regularization strength, particularly weight decay
- Clearest observation in algorithmic tasks with compact mathematical structure
Theoretical explanations:
- Circuit formation: Memorization and algorithmic circuits compete within the same network, with regularization gradually favoring the compact algorithm circuit over the expensive memorization circuit
- Representation learning: Discovering the task's mathematical structure takes time even after memorization is complete, because the gradient signals for representation quality are weak when training loss is already low
- Compression: Weight decay drives the network toward minimum description length solutions, and the algorithmic solution is more compressible than the memorization solution
- Lazy-to-rich transition: Escaping the lazy learning regime requires overcoming an energy barrier, with extended training and regularization giving the nudge toward richer, more generalizable representations
Practical implications:
- Longer training with strong regularization can yield generalization that early stopping would prevent, especially for tasks with underlying algorithmic structure
- Regularization strength critically affects whether and when grokking occurs: calibrate weight decay carefully rather than treating it as an afterthought
- More training data accelerates grokking by strengthening signals for generalizing solutions relative to memorization solutions
- Architecture choices affect grokking dynamics through inductive biases: match architecture to task structure to let efficient representation of the algorithmic solution