Grokking › Beyond Arithmetic: Grokking in Other Domains

Grokking was first discovered in modular arithmetic, but the phenomenon is not limited to these tasks. It appears across a range of algorithmic learning problems, and understanding its scope helps clarify what grokking is really about. The breadth of tasks where grokking appears also helps us understand the conditions necessary for grokking, as we can look for common properties shared by all tasks that grok.

Beyond modular arithmetic, grokking appears in polynomial evaluation (computing $f(x) \bmod p$ for polynomial $f$), sequence completion (learning rules for continuing sequences), Boolean function learning (learning XOR and parity along with other Boolean operations), and permutation group composition (learning how to compose permutations in abstract algebra). The common thread across all these tasks is the presence of underlying mathematical structure that can be represented compactly. Tasks with clean algorithmic descriptions grok; tasks without such descriptions typically do not.

This observation helps explain the mechanism: grokking requires a task with two qualitatively different solutions, memorization and an algorithm, where the algorithmic solution is substantially more compact. Without both conditions, grokking as typically described does not occur. A task with a compact algorithm but too little training data might not grok because there is insufficient signal for the algorithm to be well-constrained. A task with abundant training data but no compact algorithm might memorize efficiently but will never grok because there is no algorithmic alternative to discover.

Ongoing research examines whether grokking-like phenomena occur in language model training. The question is scientifically interesting because language has grammatical and semantic structure, along with pragmatic structure that may be learnable algorithmically rather than through memorization. If large language models undergo grokking-like transitions during training, this could explain some sudden capability improvements observed as training scales up. It would also suggest that extending training runs after loss plateaus might yield better generalization.

Several observations suggest that grokking-like behavior may occur in language models. Certain linguistic patterns, such as subject-verb agreement and long-range syntactic dependencies, show sudden improvement at specific points during training rather than gradual improvement. Models sometimes exhibit delayed emergence of grammatical understanding that appears after the model has already achieved low training loss. Continued training past apparent convergence sometimes improves specific capabilities in ways that are difficult to explain by simple loss minimization.

However, the relationship between these observations and classical grokking remains unclear. Language data lacks the clean mathematical structure of modular arithmetic, which makes it much harder to distinguish grokking from other phenomena such as capability emergence at scale or curriculum effects in the training data distribution. The training corpora used for large language models contain redundant and noisy examples whose structure is inconsistent, which makes it much harder for the clean signal about algorithmic regularities to separate from the noise. Establishing clear evidence for grokking in large language models remains an open research question, and the answer has significant practical implications for how we approach large-scale training runs.

Grokking connects to the double descent phenomenon in interesting ways. Both involve counterintuitive relationships between training procedure and generalization that violate classical statistical intuitions.

Double descent reveals that larger models can generalize better after sufficient training, contrary to the classical bias-variance tradeoff that predicts larger models will overfit more severely. Grokking reveals that longer training can improve generalization, contrary to the early stopping advice that follows from the same classical framework. Both phenomena involve neural networks finding better solutions through a combination of overparameterization and regularization than classical theory would predict.

The common thread is that modern neural networks operate in regimes where classical statistical intuitions, developed for small, well-specified models, simply break down. The interpolation threshold where models can perfectly fit all training data marks a transition to a new generalization regime where more capacity and more training can help rather than hurt. Both double descent and grokking are manifestations of this new regime, seen through different experimental lenses. Double descent is the regime of overparameterization across model scale; grokking is the regime of overparameterization across training time. Both reveal that the classical bias-variance tradeoff is not a basic law of generalization but a property of a specific class of models and training procedures that does not extend to modern deep networks.

As we discussed in the previous chapter on emergence, large language models sometimes develop capabilities suddenly and at apparently unpredictable scales. Grokking and emergence share a family resemblance: both involve sudden phase transitions in performance rather than gradual improvement. But they differ in the mechanism and the needed axis of variation.

Emergent capabilities typically appear as a function of model scale, occurring suddenly at a specific parameter count threshold. Grokking appears as a function of training time, occurring suddenly after a specific number of training steps. Both could be manifestations of the same underlying phenomenon: the coexistence of multiple solutions with different generalization properties, and the existence of phase transitions between them. Scale and training time are both axes along which the balance between solutions can shift. Some emergent capabilities in large language models may be instances of grokking that required long training runs to manifest and appear as functions of scale because larger models are also trained for more steps in absolute terms.