Mechanistic Interpretability › How Circuits Are Discovered

Discovering circuits is empirical detective work. The core methodology involves three steps: identify a clean, well-defined behavior; use ablations to find which components are necessary; and verify the components suffice to implement the behavior by studying their weights.

Good circuit analysis starts with a behavior that is as clean and minimal as possible. The classic example from early mechanistic interpretability work is "indirect object identification": given a sentence like "When Mary and John went to the store, John gave a drink to", the model should predict "Mary" as the next token. This task isolates a specific piece of linguistic knowledge (the indirect object in a double-object construction) from the full complexity of natural language understanding.

The researchers Chris Olah, Nick Cammarata, and others at Anthropic began this work by identifying behaviors in smaller transformers (two-layer models) where complete circuit analysis was tractable. Working with small, clean models before scaling is a standard methodological principle: the circuits that exist in toy models often appear in larger models too, but the full circuit is easier to characterize when the model has only a dozen attention heads rather than hundreds.

The choice of behavior matters a great deal. Behaviors that are too complex (like "solve a math word problem") decompose into many sub-tasks with independent circuits, making a unified circuit analysis intractable. Behaviors that are too simple (like "repeat the last token") may not reveal much that is interesting. The ideal behavior sits in the middle: complex enough to require non-trivial computation, simple enough that the circuit can be fully characterized. The IOI (indirect object identification) task and the greater-than task have proven productive precisely because they satisfy this balance.

A related consideration is the construction of appropriate test sets. For circuit analysis to be rigorous, you need a large, controlled set of inputs that isolate the behavior of interest from confounds. The IOI researchers created thousands of sentence templates of the form "When [Name1] and [Name2] [verb], [Name2] [verb phrase] [Name1]", varying the names systematically to ensure that their circuit discoveries held across diverse surface forms, rather than only a handful of cherry-picked examples.

As we covered in Chapter 6 (Activation Patching), causal tracing is the technique of replacing activations in a "corrupted" forward pass with activations from a "clean" forward pass to identify which components are causally necessary for correct behavior.

The procedure for circuit discovery typically works as follows. You run the model on a clean input (one that produces the correct behavior) and a corrupted input (one where the behavior breaks, such as by replacing a key name with an unrelated word). Then you systematically restore activations from the clean run into the corrupted run, one component at a time, and measure how much the correct output probability recovers.

Components whose restoration causes large recovery of the correct output are causally important. By doing this systematically across all attention heads and MLP layers, you can produce a "causal importance map" showing which parts of the network matter for a specific task. This map is often sparse: for well-defined behaviors, only a small fraction of the network's components show up as important. That sparsity is precisely what makes circuit analysis informative.

The interpretation of activation patching results requires care. Patching a component from clean to corrupted (or vice versa) measures the causal effect of that component's activation, but that effect depends on all the components downstream from it. A component might appear unimportant in isolation but become important when examined in the context of its downstream consumers. Researchers therefore combine patching with component-level weight analysis to build a complete picture.

Activation patching has one important limitation: it identifies important components but does not tell you what those components compute. Two heads might both be causally important but for different reasons. Characterizing what each component does requires additional analysis of the weight matrices.

Once you have identified the important components, you analyze their weight matrices to characterize what they compute. For attention heads, this means examining the QK and OV circuits. For MLP neurons, it means studying the input weights (which patterns activate the neuron) and the output weights (what the neuron writes to the residual stream when active).

The QK circuit can be visualized by computing the attention pattern for specific types of inputs. If a head's query-key interaction consistently produces high attention weights for positions sharing some property (such as the same token, or a specific syntactic relation), that tells you the head's selection criterion. For instance, a head that attends from each pronoun to the most recent co-referent noun would show a characteristic pattern in the QK circuit: the query direction for pronoun representations points toward the direction for noun representations that share the pronoun's gender feature.

The OV circuit can be analyzed by looking at the matrix $\mathbf{W}^V_{\ell,h} \cdot \mathbf{W}^O_{\ell,h}$ directly. If you project this matrix onto the input and output embedding spaces, you can ask: "When this head attends to a position with token X, what does it write to the residual stream?" This can be expressed as a mapping from input tokens to output logit changes.

A particularly useful projection is the "full OV circuit" mapped through the unembedding matrix. If the model's embedding matrix is $W_E$ (mapping tokens to residual stream vectors) and the unembedding matrix is $W_U$ (mapping residual stream vectors back to logit space), then the full effect of a head's OV circuit on the output distribution is captured by the product:

$$M_{\text{OV}} = W_U \cdot W^O_{\ell,h} \cdot W^V_{\ell,h} \cdot W_E$$

This product yields a $|\mathcal{V}| \times |\mathcal{V}|$ matrix, where $|\mathcal{V}|$ is the vocabulary size. Each entry $(i, j)$ tells you: "when this head attends to a token-$j$ position, how much does it promote token $i$ in the final output logits?" If the matrix is approximately diagonal (entry $(i, i)$ dominates row $i$), the head implements a copy operation. Off-diagonal structure indicates a more complex transformation, such as mapping attended tokens to related but different output tokens (for example, a head that reads a country name and promotes its capital city in the output logits).

Weight analysis is most tractable in small models, because the weight matrices are small enough to visualize and reason about directly. In larger models, researchers use dimensionality reduction, sparse approximations, or targeted projections to identify the key directions.

Path patching is a refinement of activation patching that lets you study individual edges rather than individual nodes. Standard activation patching asks "does this component matter?" Path patching asks "does information flowing from component A to component B matter?"

To path-patch from component A to component B, you corrupt the edge between them specifically: run the corrupted input normally through all layers, but replace only the input to B with the clean-run value of A's output. If this specific edge is critical, the target behavior recovers. If not, the edge is not part of the circuit even if both components individually are.

Path patching enables more precise circuit characterization. You can confirm that "heads A and B are both important" and that "A feeds directly into B via the residual stream in a way that is causally necessary." When combined with weight analysis, path patching lets you build a full circuit description: here are the nodes, here are the edges, and here is the weight-level explanation for each edge's function. That complete description is what distinguishes a mechanistic understanding from a mere list of important components.

The full workflow of behavioral isolation, activation patching, weight analysis, and path patching forms a convergent methodology. Each step builds on the previous: behavioral isolation identifies what to study; activation patching narrows the field to important components; weight analysis characterizes what those components do; and path patching verifies the edges connecting them. No single step suffices, but together they allow researchers to make strong, empirically grounded claims about specific circuits.