Grokking › Grokking Mechanism Theories

Why does grokking happen? Several theories have been proposed, each giving partial insight into this puzzling phenomenon. These theories are not mutually exclusive. They illuminate different aspects of the same underlying process, revealing the complexity of neural network learning and how multiple forces shape the evolution of representations during training. Understanding these theories helps explain grokking and builds a richer mental model of how neural networks learn in general.

The central puzzle that any theory must explain is this: the memorization solution and the generalizing solution are both compatible with achieving low training loss. So why does the network prefer the memorization solution initially, and why does it eventually switch to the generalizing solution? The answer must involve something that changes over training time, something that tilts the balance from one solution to the other. The four major theories each identify a different "something": circuit formation stresses the relative cost of two parallel circuits, representation learning focuses on the quality of learned abstractions, the compression perspective invokes information-theoretic arguments about description length, and the lazy-to-rich framework draws on theoretical analysis of infinite-width network dynamics.

Each theory gives a useful lens for understanding grokking, and together they paint a picture of the phenomenon that is richer than any single explanation could give. As you read through these theories, notice how they all converge on a common theme: grokking is driven by the interaction between two competing forces, the gradient descent pressure to minimize training loss and the regularization pressure to minimize parameter magnitude, and the temporal dynamics of that competition determine when and whether generalization emerges.

One influential explanation comes from mechanistic interpretability research. According to this view, neural networks can stand for solutions in multiple ways. One is a memorization circuit that stores input-output pairs explicitly, in effect acting as a lookup table. Another is an algorithm circuit that computes the correct answer using the underlying mathematical structure of the task. These two circuits can coexist within the same network, with their relative strengths determining the model's behavior at any given point in training.

Think of it as two employees working on the same problem. The first employee memorizes all the specific cases they have seen, responding instantly to known situations but failing completely on unfamiliar ones. The second employee understands the underlying rules and can solve any problem, but takes longer to develop that understanding. Early in training, the memorizer's approach is faster to learn, so it dominates. But maintaining a detailed lookup table requires more resources than applying general rules, and over time, the rule-follower's approach becomes increasingly efficient by comparison. When regularization penalizes the resources consumed by the memorizer, the balance eventually tips toward the rule-follower.

For modular addition, the algorithmic solution is numbers as points on a circle. Since modular arithmetic wraps around, the network can compute their sum using trigonometric relationships. Addition modulo $p$ has inherent circular structure, resembling how a clock cycles from 12 back to 1. By embedding numbers as points on a circle, the network can exploit this geometric structure directly. Each number $k$ maps to a point on the unit circle according to:

$$\text{embedding}(k) = \left(\cos\left(\frac{2\pi k}{p}\right), \sin\left(\frac{2\pi k}{p}\right)\right)$$

where:

• $p$: the prime modulus defining the arithmetic system
• $k$: the number being embedded, an integer from 0 to $p-1$
• $\text{embedding}(k)$: a two-dimensional vector representing $k$ as a point on the unit circle
• $\cos\left(\frac{2\pi k}{p}\right)$: the x-coordinate of the point, the horizontal position on the circle ranging from -1 to 1
• $\sin\left(\frac{2\pi k}{p}\right)$: the y-coordinate of the point, the vertical position on the circle ranging from -1 to 1
• $\frac{2\pi k}{p}$: the angle in radians that maps number $k$ to its position on the circle

To understand this formula intuitively, imagine a clock face divided into $p$ equal sections rather than 12. This embedding places each number at a unique position on the unit circle, evenly spaced around the circumference. The number 0 sits at the rightmost point (the 3 o'clock position), and as $k$ increases, you move counterclockwise around the circle. Dividing the full circle angle $2\pi$ by $p$ gives the angular spacing between consecutive numbers. This representation captures the wraparound property because when numbers are embedded on a circle, adding two numbers becomes geometrically equivalent to adding their angles.

Why does this formula make sense? Notice that after $p$ steps, $k = p$ gives $\cos(2\pi) = 1$ and $\sin(2\pi) = 0$, which is the same as $k = 0$. The circular embedding therefore naturally captures the cyclic structure: the number after $p-1$ wraps back to 0, just as position $p$ on the circle returns to the starting position. No special-casing of the wraparound boundary is needed; the circular geometry handles it automatically. This automatic handling of the boundary case is why the circular solution is more compact than the memorization solution, which must store the boundary cases explicitly.

The key insight from the circuit formation hypothesis is that the two circuits are not competing for the same parameters but coexisting as parallel internal representations that contribute to the final output. Early in training, the memorization circuit's contributions dominate because it was easier to form. But over extended training with weight decay, the relative magnitudes shift. The weight decay penalty accumulates more heavily on the memorization circuit's large, specific parameters than on the algorithm circuit's compact, reusable parameters. Eventually, the algorithm circuit's contributions become large enough relative to the memorization circuit's contributions that the model's external behavior changes.

The visualization shows how numbers 0 through 16 are evenly distributed around the unit circle. The highlighted points (5, 7, and 12) show how addition works geometrically: adding 5 and 7 corresponds to combining their angular positions, resulting in the position for 12. The circular structure that the network must discover during grokking to reach true generalization is visually apparent: numbers are not arranged on a number line but on a circle where the wraparound is built into the geometry.

This circular representation captures the wraparound property of modular arithmetic. After reaching $p-1$, the next value 0 appears adjacent on the circle, making the structure cyclic rather than linear. The circular embedding ensures numbers close in modular arithmetic are also close in the embedding space, letting the network to learn smooth, generalizable functions instead of storing separate parameters for each input.

Consider what happens when we add $a$ and $b$ in this circular representation. Each number corresponds to a specific angle around the circle, and their sum corresponds to the angle you reach by first rotating to angle $a$ and then rotating by an additional angle $b$. The modular wraparound happens automatically because angles naturally wrap around: rotating by more than $2\pi$ radians brings you back around the circle. Mathematically, this angular addition is:

$$\begin{aligned} \text{angle}(a + b) &= \frac{2\pi a}{p} + \frac{2\pi b}{p} && \text{(add the individual angles)} \\ &= \frac{2\pi(a + b)}{p} && \text{(factor out common terms)} \end{aligned}$$

where:

• $\text{angle}(a + b)$: the resulting angle after adding $a$ and $b$ on the circle
• $\frac{2\pi a}{p}$: the angle corresponding to number $a$, its position on the circle
• $\frac{2\pi b}{p}$: the angle corresponding to number $b$, its position on the circle
• $\frac{2\pi(a + b)}{p}$: the combined angle, which naturally wraps around at $2\pi$ and captures the modular wraparound

This geometric operation can be computed using attention mechanisms that learn to compose these rotations through the transformer's query-key-value operations. The key advantage of this approach is that it requires only a constant number of parameters regardless of the number of possible input pairs, since the same rotation operation applies to all inputs. This compactness is why the algorithm circuit is more efficient than the memorization circuit, and why regularization eventually favors it.

During training, four stages unfold:

1. The memorization circuit forms first because it is the fastest path to low training loss. Gradient descent quickly finds a way to map each seen training example to its correct output by storing pattern-specific parameters.
2. The algorithm circuit develops slowly in parallel: the correct representations and computations require coordinated changes across multiple layers, which takes many more gradient steps to reach.
3. Weight decay gradually penalizes the memorization circuit more heavily than the algorithm circuit, because memorization requires storing many large parameters while the algorithm is more compact.
4. Eventually, the algorithm circuit becomes strong enough to dominate, and generalization emerges.

This view explains why weight decay is needed for grokking: it gives the asymmetric pressure that eventually favors compact, generalizing solutions over sprawling memorization. Without weight decay, both solutions have the same training loss, and the memorization solution, being easier to find, persists indefinitely. The circuit formation hypothesis makes a testable prediction: if we could directly examine the network's internal representations during training, we should see evidence of both circuits forming early, with the algorithm circuit's relative strength gradually increasing during the plateau phase before the grokking transition. Follow-up work using Fourier analysis of learned embeddings has provided strong empirical support for exactly this prediction.

A related perspective focuses on how representation learning drives grokking. Rather than framing the issue as a competition between two circuits, this view stresses that the model must discover that numbers in modular arithmetic have circular structure. The quality of the learned representations determines whether generalization is possible at all. Even if the model has learned some circuit implementing modular addition, if its internal representations of numbers do not capture the circular structure, that circuit will fail on unseen inputs.

The representations the model needs to learn have specific properties:

• Numbers close in modular distance should be close in embedding space, so the model can extrapolate smoothly between them.
• The embedding should capture the cyclic nature of modular arithmetic, where 0 is close to $p-1$.
• Addition should correspond to a composable operation in this space, implemented via rotation.

Learning such structured representations takes time for a specific reason: random initialization places numbers without real structure, and the gradient signals for improving representation quality are weak when memorization already reaches low training loss. The model has no immediate incentive to reorganize its embeddings into a better structure while the memorization solution is working perfectly. From gradient descent's perspective, if the current parameters produce zero training loss, why change them? The only pressure comes from weight decay, which quietly erodes the large parameter configurations and creates gradual incentive to find more compact representations.

Think of this as the difference between a student who learns vocabulary by rote versus one who learns word roots and etymology. The rote learner can answer flashcard questions immediately but struggles with new words. The etymology learner needs more time upfront to build the conceptual framework, but that framework generalizes. In early training, the rote approach is faster to implement, so that is what gradient descent discovers first. Only over extended training, with regularization giving pressure toward compactness, does the more structured approach emerge.

From the representation-learning perspective, gradient descent traverses a loss surface containing many solutions, and its path determines which solutions it encounters first. Memorization lies close to random initialization in parameter space, which is why it appears early. The algorithmic solution lies in a different region and takes longer to reach. The intervening loss landscape may contain local minima, saddle points, and regions of slow progress. The network can therefore spend a long time approaching the algorithmic region before crossing a threshold at which that solution becomes locally dominant.

Only after extended training do the cumulative effects of weight decay and gradient updates align the representations with the mathematical structure of the task. The alignment is gradual and largely invisible in the training curves. Its effects appear in test accuracy only after a threshold of structural quality is crossed, which makes grokking look sudden from the outside. The plateau phase can therefore contain internal reorganization that is absent from the observable metrics.

Information-theoretic views frame grokking as a compression problem. A memorizing solution has high description length: it must store each training example separately as a specific set of parameter values. An algorithmic solution has low description length: it only needs to store the rules for addition, which can be expressed compactly in terms of circular embeddings and rotation operations.

Think of the description length difference concretely. To store the answer to each of the 4,704 training examples individually, you need at minimum $4704 \times \log_2(97) \approx 31,500$ bits of information. To store the circular embedding for $p=97$ and the rotation operation, you need information proportional to the size of the embedding (roughly 200 floating-point numbers, or about 6,400 bits). The algorithmic solution is roughly five times more compact in information-theoretic terms, even before accounting for the fact that the algorithmic solution covers all 9,409 possible inputs rather than just the 4,704 training examples.

Weight decay acts as a description length penalty, systematically favoring solutions that use fewer bits to stand for their knowledge. The network starts by memorizing because memorization is the fastest way to reduce training loss. But as training continues, the regularization pressure steadily drives the network toward more compressed representations. Grokking occurs when the compressed algorithmic solution finally reaches competitive training loss.

This connects to the principle of Occam's razor and minimum description length (MDL) in learning theory: given two hypotheses that explain the data equally well, prefer the simpler one. Weight decay operationalizes this preference by penalizing the complexity of the model's parameters. The MDL perspective makes a specific prediction: grokking should occur more readily in tasks where the algorithmic solution is more compact than the memorization solution. This prediction has been confirmed empirically. Modular arithmetic groks because its circular structure is highly compressible. Random Boolean functions, where the output has no structure correlated with the input, do not grok because there is no compact description of the function; memorization is the only available solution. The presence or absence of compressible structure in a task is therefore a key predictor of whether grokking will occur.

Another framework for understanding grokking distinguishes between lazy and rich learning regimes. These concepts come from theoretical analyses of infinite-width neural networks, but they give useful intuitions for finite networks as well.

In the lazy regime, the network makes minimal changes to its initial representations during training. It in effect performs a form of kernel regression using features from the initial random network, without substantially adapting those features to the task. Lazy learning is computationally efficient: small parameter changes suffice to reach low training loss. But the solutions found in the lazy regime may generalize poorly because they are anchored to the idiosyncratic features present at initialization. For modular arithmetic, a lazy solution would use whatever quasi-circular features happen to be present in the random initialization and build a lookup table on top of them. This solution reaches low training loss but does not generalize because the features are not truly circular.

In the rich regime, representations change substantially during training, adapting to the structure of the task. The network develops features that are specifically useful for the problem at hand, rather than relying on whatever features happened to be present at initialization. Rich learning requires more gradient steps and larger parameter changes, but it produces representations with better inductive structure. For modular arithmetic, rich learning means the embeddings reorganize to lie on a circle, and the attention mechanisms implement angular addition rather than a lookup table.

Grokking may be a phase transition from lazy to rich learning. Early training operates in the lazy regime, where the network memorizes using representations close to initialization. Continued training with regularization can push the network into the rich regime, where representations restructure to capture the underlying algorithm. This transition takes time because escaping the lazy regime requires overcoming an energy barrier in the loss surface. The network must temporarily reorganize its representations in ways that do not initially improve training performance before arriving at representations that support generalization.

The lazy-to-rich transition framework connects grokking to a broader theoretical question about what kinds of learning occur in neural networks. A network in the lazy regime exploits random initialization features that work for the training examples it has seen. A network in the rich regime has learned the task structure. Grokking describes a transition from memorizing in the lazy sense to learning the underlying rule in the rich sense. This distinction between shallow performance and structural understanding is central to grokking.