Grokking › The Grokking Phenomenon

Traditional machine learning theory suggests a model's generalization ability is determined early in training. The classical perspective views the training process as a race between fitting the data and overfitting: you want to stop at the sweet spot where the model has learned real patterns but has not started memorizing noise. Once the model starts overfitting, the gap between training and test performance should widen monotonically. The textbook advice is unambiguous: stop training when validation loss starts increasing, or you will hurt generalization. This guidance emerges from decades of experience with classical statistical models, and it is backed by a solid theoretical framework called the bias-variance tradeoff. Models that fit the training data too closely pick up noise instead of signal, and their performance on new data suffers as a result.

Grokking violates this expectation. Rather than worsening steadily with continued training, the relation between memorization and generalization can change over time. Networks can maintain multiple internal solutions, with regularization and continued optimization shifting which one dominates. A model can appear completely overfit while slowly building the basis for generalization. It has memorized the training data, but its training metrics do not reveal the internal reorganization that may later support generalization.

The key insight here is that "overfitting" as measured by the gap between training and test accuracy captures the model's current behavior but says nothing about the trajectory of its internal representations. A model that has memorized the training data is not frozen in that state. Regularization pressure continues to reshape its parameters, and that reshaping can eventually cross a threshold where a generalizing solution becomes dominant. The standard early stopping criterion monitors the surface behavior without seeing the internal reorganization, which is why it fails to anticipate grokking.

A typical grokking training curve exhibits three distinct phases, each with its own characteristic signature in both training and test metrics.

1. Memorization phase: Training accuracy quickly reaches 100% and training loss approaches zero. The model has learned to reproduce every training example correctly, but test accuracy remains at or near chance level. The model appears to have fully overfit the training data, and by standard criteria, training should stop here.
2. Plateau phase: Validation accuracy remains at chance level for an extended period spanning thousands to millions of steps. From the outside, the model appears completely overfit with no hope of recovery. Standard early stopping criteria would halt training here, and the practitioner would move on. But internally, the model's representations are slowly reorganizing in response to continued regularization pressure.
3. Grokking phase: Validation accuracy suddenly jumps, quickly approaching training accuracy. The model transitions from near-random predictions on test data to near-perfect predictions, often within a small fraction of the total training time. The transition appears abrupt and can feel counterintuitive: nothing in the training procedure changed, yet performance improved sharply.

The delay between memorization and generalization can be enormous, far exceeding what any standard learning curve would predict. In some experiments, models trained for 10,000 steps show no generalization, but by step 100,000 or 1,000,000, they suddenly reach near-perfect test accuracy. The key insight is that generalization and memorization can coexist in the same network, with the generalizing solution emerging slowly beneath the surface while the memorizing solution dominates the observable behavior. The network develops two parallel representations: one stores training examples by rote, another gradually discovers the underlying task structure. Eventually, structural understanding becomes dominant, and that dominance expresses itself in the sudden jump in test accuracy.

Understanding why this matters beyond academic curiosity requires appreciating what it tells us about the nature of learning in neural networks. If grokking is possible, then apparent overfitting may not be the end state we thought it was. Models that appear to have finished learning may still be in the process of discovering compact, generalizable solutions. This has implications for training schedules, early stopping strategies, and our basic understanding of what happens inside neural networks during optimization. It also raises provocative questions about large language models: do they grok certain capabilities during pretraining? Is there a class of linguistic competencies that only emerge after extended training past apparent convergence? These questions remain open, but grokking gives the conceptual framework for thinking about them.