Grokking › Grokking Phase Transitions

Grokking exhibits sharp phase-transition behavior characteristic of many complex systems in physics and mathematics. Generalization can appear over a relatively small number of training steps compared with the total training time. This distinguishes grokking from standard learning curves, where test accuracy typically improves smoothly alongside training accuracy.

The suddenness of the transition is itself informative. It suggests that grokking is a qualitative change in the network's internal organization rather than a simple quantitative accumulation of knowledge. Something structural changes when grokking occurs, beyond a small improvement in a few parameters. The network reorganizes from one kind of solution to another, and that reorganization, when it happens, happens fast. This is a key distinguishing feature: if grokking were merely a very slow version of ordinary learning, we would expect test accuracy to improve slowly and continuously. The fact that it jumps suddenly implies a threshold effect, where the algorithmic solution must cross some minimum quality threshold before it can dominate the memorization solution's contributions to the output.

Think of this like a phase transition in physics. When water cools toward 0 degrees Celsius, most properties change gradually with temperature. But at exactly 0 degrees, a sharp transition occurs: liquid water becomes solid ice. The system's structure reorganizes fundamentally. Grokking is analogous: during the plateau phase, internal reorganization is happening gradually, but the observable behavior (test accuracy) only changes when the reorganization crosses a threshold, at which point the transition appears sudden from outside. Statistical physics has developed advanced tools for characterizing and predicting such phase transitions, and some researchers have begun applying these tools to grokking, searching for mathematical signatures of the transition that can be computed from the model's state before the transition occurs.

Several factors influence when grokking occurs, and understanding these factors helps predict and control the transition.

Weight decay strength is the primary control knob. Too little weight decay means the model memorizes without sufficient pressure to reorganize toward generalization. Too much weight decay interferes with learning entirely, preventing the model from finding any low-loss solution. A necessary range exists where weight decay is strong enough to favor the algorithmic solution but not so strong that learning becomes impossible. Within this range, grokking occurs reliably, and increasing weight decay within this range generally accelerates grokking by increasing the pressure on the memorization circuit.

Training data fraction affects the memorization-generalization tradeoff in a more subtle way. More training data means more examples for the network to memorize, which increases the cost of the memorization solution while keeping the cost of the algorithmic solution roughly constant. More data therefore accelerates grokking by widening the gap between memorization cost and algorithm cost. Less training data makes memorization cheaper relative to the algorithm, delaying or preventing grokking. This is a somewhat counterintuitive result: in standard machine learning, we expect more data to always help, and it does here too, but through a specific mechanism related to the cost differential between the two solutions rather than through improved signal quality.

Model capacity plays a complex, non-monotonic role. Very small models may lack the capacity to stand for either memorization or the algorithm. Very large models memorize so easily that regularization cannot effectively distinguish between the two solutions. An intermediate model capacity is optimal: it must work to memorize, allowing regularization to favor the algorithm. The sweet spot for grokking exists when the model is large enough to stand for both solutions but not so large that memorization becomes trivially cheap.

The relationship between weight decay and grokking speed illustrates the phase transition nature of the phenomenon. At low weight decay, the model memorizes and stays memorized. As weight decay increases past a necessary threshold, grokking appears and happens progressively faster. Beyond some optimal range, strong regularization can interfere with learning entirely, preventing even memorization. The necessary range where grokking occurs reliably is task-specific and model-specific, which is why careful hyperparameter selection matters. You cannot simply apply any positive weight decay and expect grokking; the weight decay must be strong enough to matter but not so strong that it overwhelms the task signal.

The optimizer choice influences grokking dynamics in ways that go beyond simple learning rate effects. AdamW, which uses decoupled weight decay, tends to produce more reliable grokking behavior than standard Adam with L2 regularization implemented as a loss term. This distinction matters: in standard Adam, weight decay is effectively scaled by the adaptive learning rates, which means parameters with high gradient variance receive weaker regularization. In AdamW, weight decay acts uniformly on all parameters. This gives cleaner pressure toward compactness.

The decoupling in AdamW is mathematically straightforward but practically important. Standard Adam with L2 regularization computes the gradient as $g_t = \nabla \mathcal{L} + \lambda \theta_t$, then applies adaptive scaling to the entire gradient including the regularization term. This means the regularization effect is weaker for parameters with high gradient variance (those with large second moment estimates). AdamW instead applies weight decay directly to the parameters after the adaptive update step: $\theta_{t+1} = \theta_t - \alpha \hat{m}_t / \sqrt{\hat{v}_t} - \alpha \lambda \theta_t$. The weight decay term is not scaled by the adaptive learning rate, so it applies uniformly. For studying grokking, this uniformity ensures that the regularization pressure on the memorization circuit is predictable and controllable.

Learning rate also matters. Higher learning rates can accelerate the phase transition once it begins, letting the network to rapidly reorganize from memorization to algorithmic solution. However, too-high learning rates cause instability that can prevent grokking entirely by disrupting the delicate process of representation reorganization. Lower learning rates lead to slower but more stable grokking. There is evidence that learning rate schedules, particularly warmup followed by cosine decay, can improve grokking reliability by letting stable memorization early in training followed by gradual pressure toward generalization.

The interaction between learning rate and weight decay is particularly important. Both affect the effective scale of parameters, and their ratio determines how aggressively the optimizer pushes toward compact solutions. Scanning both hyperparameters together, rather than independently, produces more systematic understanding of grokking behavior. Practitioners working with tasks where grokking dynamics may be needed should treat both learning rate and weight decay as primary hyperparameters to tune jointly rather than sequentially.

We can quantify the sharpness of the grokking transition by examining the rate of change of test accuracy over training. Rather than looking at raw accuracy values, which tell us where the model is, examining the derivative reveals how quickly the model is improving at each point in training. This derivative perspective helps distinguish between gradual improvement, where accuracy increases smoothly over time, and the sudden transitions characteristic of grokking, where almost all improvement is concentrated in a narrow window.

The derivative of test accuracy with respect to training step is:

$$\frac{d(\text{accuracy})}{d(\text{step})}$$

where:

• $\text{accuracy}$: the test accuracy at a given point in training, a value between 0 and 1
• $\text{step}$: the training step number, counting gradient updates or epochs
• $\frac{d(\text{accuracy})}{d(\text{step})}$: the derivative showing how quickly accuracy changes per training step, representing the instantaneous rate of improvement measured in accuracy units per step

This derivative measures the instantaneous rate at which test accuracy improves during training. A large positive value indicates rapid improvement: the model is quickly getting better at the task. Values near zero indicate periods of stagnation where accuracy remains roughly constant despite continued training. During grokking, the derivative shows a distinctive pattern: it stays near zero during the plateau phase, spikes sharply during the transition, then returns to near zero after full generalization is achieved.

These peaks in the derivative characterize the transition from memorization to generalization. The sharper the spike, the more abrupt the transition from one regime to the other. This concentration of improvement within a narrow window of training steps is what makes grokking visually striking on learning curves. The mathematical signature of grokking is high eventual accuracy paired with this distinctive pattern of delayed-then-rapid improvement.