Activation Patching: Causal Tracing in Neural Networks

Imagine you want to know which part of a complex machine is responsible for a specific behavior. You could tear it apart and examine every component individually, but a smarter approach is to surgically swap one component at a time with a different version and observe whether the behavior changes. If replacing component X changes the output, X is causally involved. If replacing it changes nothing, X is irrelevant to that behavior. This is the core idea behind activation patching, and it turns out to be exactly the right tool for interrogating the inner workings of large language models.

Activation patching is a technique for locating where in a neural network specific information is stored and processed. Instead of analyzing weights or gradients in isolation, activation patching runs two forward passes through the model and then transplants activations from one pass into the other. By selectively patching at different positions, layers, and component types, you can trace the causal path through which information flows to produce a given output. The result is a rich, interpretable map of computational responsibility: a picture of which parts of the model matter for which behaviors.

As we discussed in the Probing Classifiers and Probing Layers chapters, probing tells you whether information is present in a representation, but it doesn't tell you whether that information is used by the network to compute its output. A probe might detect entity type encoded in layer 10's residual stream, but the model might reach its final prediction through a completely different computation that never reads that representation. Activation patching closes this gap. It provides causal evidence that a particular activation, at a particular position, at a particular layer, lies on the computational path from input to output. This distinction between correlation and causation is fundamental: probing is diagnostic, but activation patching is interventional.

The difference matters enormously in practice. Suppose a researcher uses a probe to show that subject entity type is decodable from layer 5's representation. Does this mean layer 5 is responsible for entity-based reasoning? Not necessarily. The entity information might be present because some earlier layer wrote it there, but the downstream computation that uses entity information for the final prediction might read from a completely different layer. Activation patching tests this directly: if you intervene on layer 5's activation and transplant the right information, does the output change? Only if it does can you conclude that layer 5 is causally, not just statistically, involved.

This technique was popularized in the mechanistic interpretability community through work by Meng et al. on factual associations in GPT-style models (the ROME paper) and through Elhage et al.'s circuit analysis work at Anthropic. It has since become a foundational method for understanding how transformers implement algorithms and store knowledge, and it underlies the most precise circuit-level analyses of transformer behavior that have been published.

Activation patching works by carefully comparing what happens when a model processes two different inputs: a "clean" input that produces the behavior you want to understand, and a "corrupted" input that disrupts that behavior. The methodology is deceptively simple to describe but requires careful thought to execute and interpret correctly.

The procedure begins by defining two inputs that differ in a meaningful way. In the classic factual association setting from the ROME paper, the clean prompt might be "The Eiffel Tower is located in" and the corrupted prompt might be "The Colosseum is located in". The model answers the clean prompt with "Paris" and the corrupted prompt with "Rome". Now you want to know: which activations during the clean run are responsible for the model outputting "Paris"?

The patching experiment works as follows. You run the model on both inputs and cache all intermediate activations. Then you run the model again on the corrupted input, but at a specific position and layer, you replace the activation from the corrupted run with the corresponding activation from the clean run. You observe whether the model's output shifts back toward "Paris". The key intuition is: if swapping that single activation changes the output toward the clean prediction, then that activation was carrying information that the model needs for the clean behavior.

More precisely, the three runs are:

• Clean run: Process input $x_{\text{clean}}$, saving all activations $\{a^{l, p}_{\text{clean}}\}$ for every layer $l$ and token position $p$.
• Corrupted run: Process input $x_{\text{corrupt}}$, saving all activations $\{a^{l, p}_{\text{corrupt}}\}$.
• Patched run: Process $x_{\text{corrupt}}$ but at a specific (layer $l^*$, position $p^*$), replace $a^{l^*, p^*}_{\text{corrupt}}$ with $a^{l^*, p^*}_{\text{clean}}$. All other activations are computed normally from the corrupted input.

If the patched run's output recovers toward the clean output, the patched activation was causally important. If the output doesn't change, that activation carries no information the model is using for this specific behavior.

To quantify how much a patch recovers the clean behavior, researchers typically track the probability or log-probability of the clean answer token. Let $P_{\text{orig}}$ be the model's probability of the correct answer under the corrupted run (the baseline without any patching), and let $P_{\text{clean}}$ be the probability under the unmodified clean run. For a patched run where the activation at position $(l^*, p^*)$ has been replaced, the probability of the target token is $P_{\text{patched}}$.

The normalized patch effect is defined as:

where:

• $l^*$: the layer index being patched
• $p^*$: the token position index being patched
• $P_{\text{orig}}$: probability of the clean answer under the unpatched corrupted run, serving as the zero-recovery baseline
• $P_{\text{clean}}$: probability of the clean answer under the clean run, serving as the full-recovery ceiling
• $P_{\text{patched}}$: probability of the clean answer after transplanting the activation at $(l^*, p^*)$ from the clean run into the corrupted run

A value of 1.0 means the patch fully restores the clean behavior: transplanting just this one activation was sufficient to recover the correct answer. A value of 0.0 means the patch has no effect at all. Values between 0 and 1 indicate partial recovery, where the patched site contributes some information but not all that is needed to fully recover the behavior.

This normalization is essential for making results comparable across different prompts and models. Without it, a small absolute probability gap (say, 0.03 to 0.05) and a large one (say, 0.10 to 0.85) would be on incomparable scales. The normalized effect always lies in $[0, 1]$ when the patch cannot harm the corrupted run beyond its baseline, though values slightly outside this range can occur due to nonlinearities in the model.

Probability is preferred because probability is preferred over logit difference or some other metric. Probability is bounded and interpretable, and it directly reflects what matters in practice: the model's confidence in the correct answer. Log-probability is sometimes used instead because it has better numerical properties when probabilities are very small or very large, but normalized probability works well for most analyses.

The granularity of what you patch matters enormously, because each choice asks a fundamentally different question about the model's computation. Common choices include:

• Residual stream vectors: The main hidden state at each position after each layer. Patching a residual stream vector at position $p$ and layer $l$ tests whether the total information at that point in the computation is causally important for the behavior.
• Attention head outputs: The output of a specific attention head, which is a vector in the residual stream dimension that gets added to the stream. Patching an individual head output isolates that head's contribution, asking whether this specific head's reading-and-writing operation is causally involved.
• MLP layer outputs: The output of the feed-forward sublayer at a specific position. Patching this substitutes the MLP's transformation while leaving the attention contribution from the corrupted run in place.
• Attention patterns: The attention weight matrix itself, independent of the value vectors. Patching this tests whether the routing of information is causally important, separate from the content of what is being routed.
• Key, query, or value projections: Even finer-grained components within a single attention head, allowing you to ask whether a head's effect comes from what it attends to, what information it writes, or the combination.

Each choice asks a different question. Patching residual streams gives a coarse view of which layer and position matter. Patching individual heads or MLP layers provides finer attribution. Patching attention patterns specifically tests the routing computation. The choice should match the question you are trying to answer, and in practice researchers often use multiple granularities in sequence: start coarse with residual stream patching to identify important (layer, position) pairs, then drill down with component-level patching to understand the mechanism.

The broadest use of activation patching is causal tracing, which systematically sweeps all positions and layers to produce a complete map of where information flows through the network. Rather than patching one site at a time and moving on, causal tracing treats the patching experiment as a measurement tool and applies it exhaustively across the model's spatial and temporal dimensions.

In causal tracing, you run the patching experiment for every combination of layer $l$ and token position $p$. The result is a two-dimensional matrix of patch effects, with token positions on one axis and layers on the other. Cells with high effect values indicate that the activation at that (layer, position) coordinate lies on the causal path for the behavior.

This matrix is typically visualized as a heatmap, and the resulting picture is often remarkably structured. When Meng et al. ran causal tracing on GPT-style models for factual associations, they found a striking pattern: the strongest effects were concentrated in the early-to-middle MLP layers at the token position corresponding to the last token of the subject entity. When the model is completing "The Eiffel Tower is located in", the critical information appears to flow through MLP layers processing the "Tower" token position.

This finding suggested that factual associations are stored in MLP weights as key-value pairs, with the subject token's accumulated representation acting as the lookup key. The MLP receives the subject's representation, matches it against patterns learned during pretraining, and retrieves the associated fact. This interpretation was further supported by the ROME paper's direct manipulation of MLP weights to update factual associations. We will revisit it in the Mechanistic Interpretability chapter when we examine circuits in detail.

What makes causal tracing so informative is that it identifies which layers matter and reveals the structure of information flow. The heatmap shows not one bright spot but typically a characteristic shape: an early region where the fact is first "found" in MLP layers at the subject position, and a late region where information arrives at the last token position just before the output is generated. Understanding this shape is understanding the factual retrieval circuit at a high level.

A common choice for creating the corrupted input in causal tracing is adding Gaussian noise to the embeddings of the subject tokens, rather than substituting a different subject entity entirely. This approach has a practical advantage: it corrupts the model's ability to retrieve the fact without changing the prompt's syntactic structure. If you use a different subject (France vs. Germany), the model might answer differently for reasons unrelated to the factual lookup. Different subjects have different tokenization lengths, different frequencies in training data, and different syntactic contexts. A substantially different subject might create confounds that make the causal trace harder to interpret.

Adding noise directly to token embeddings produces a controlled corruption. The subject tokens are still present in the input as tokens, but their embedding representations are scrambled, preventing the model from recognizing the entity. This isolates the effect of "knowing" the subject from any structural differences between two different prompts. The rest of the prompt remains identical, so the syntactic context and token positions are preserved.

Let $e_i$ be the embedding of subject token $i$ before adding positional information. The corrupted embedding is:

where:

• $e_i$: the original embedding vector for token $i$
• $\epsilon_i$: Gaussian noise sampled independently for each subject token
• $\sigma$: noise magnitude, typically set to 3 times the empirical standard deviation of the embedding vectors

The noise magnitude $\sigma$ requires calibration. Too small, and the model can still recognize the entity, giving a weak baseline disruption. Too large, and the corrupted input becomes so far out of distribution that the model's behavior is unpredictable and the causal trace becomes noisy. The choice $\sigma = 3\sigma_e$, where $\sigma_e$ is the empirical standard deviation of the embedding distribution across a sample of real tokens, provides a reliable disruption while remaining in a regime where the model continues to process the input coherently.

An important subtlety: when you patch the clean activation at a particular layer and position, you are restoring it into a context where the surrounding activations came from a noise-corrupted input. The attention mechanism at later layers will attend across positions, mixing the patched (clean) activation at one position with corrupted activations elsewhere. This is not a flaw but a feature: the fact that a single clean activation can still recover the behavior despite the mismatched context argues strongly for that activation's causal role.

A key insight from causal tracing is that information doesn't just sit statically in one place. Once a critical activation is established at some layer and position, it propagates forward through later layers. Downstream attention heads read from earlier residual streams and copy information to later positions. This means that in the causal trace matrix, you typically see both "early sites" where information first appears and "late sites" where it arrives at the final prediction position.

The ROME paper introduced explicit terminology for this distinction. Early sites are the first layers where the patch has a large effect, typically middle MLP layers at the subject position. These are the locations where the model first looks up or constructs the factual information. Late sites are the final layers before the prediction, typically at the last token position where the output is generated. Information flows from early sites through attention layers to late sites, where it is used to generate the next token.

Understanding this propagation path is key to understanding the full circuit. The early MLP sites are the "storage" locations: they encode factual associations in their weights and retrieve them given the right activations. The late attention sites are the "routing" mechanism: they read from earlier positions and bring the relevant information to the final prediction position. A complete mechanistic account of factual recall needs both halves of this story.

This propagation structure also explains why residual stream patching at a late layer and position often shows high effects even if the computation happened much earlier. The information has been propagated forward by attention. If you patch the residual stream at layer 11, token position 7, you are providing the network with everything it had during the clean run right before the output, so of course the output recovers. This is a high effect but an uninformative one: it is too close to the output to tell you where the computation happened. This is why causal tracing typically focuses on the earliest high-effect site as the true computational source, rather than the highest-effect site overall.

Activation patching is particularly powerful for locating where specific types of information are encoded in a model, and the granularity of this localization has proven useful for guiding subsequent mechanistic work.

In factual recall tasks, a prompt like "The Eiffel Tower is located in" contains at least two semantic components worth distinguishing: the subject ("The Eiffel Tower") and the relation ("is located in"). The object ("Paris") is what the model must retrieve. Causal tracing can reveal which component's processing is most critical for retrieval, which in turn informs theories of how factual knowledge is organized.

The intuition behind this decomposition is that different components of a fact may be processed at different locations and times in the transformer. The subject identifies which entity we are asking about. The relation specifies what type of information to retrieve about that entity. A model that has learned factual associations might process the subject to construct an entity representation, then use the relation to select which fact to retrieve, and finally generate the object as output.

Empirically, researchers have found that for factual associations:

• Subject tokens, especially the last subject token, carry the most critical information in early-to-middle layers, particularly in MLP sublayers. The last subject token aggregates information about the entire subject through causal attention and is the natural site for entity-level processing.
• Relation tokens have smaller effects in most cases, though they can be important for specific relation types where the relation itself disambiguates among multiple plausible answers.
• The final token position sees high effects in the last few layers, showing the model assembling the answer right before output, which is the late-site propagation described above.

Why does the last subject token carry so much weight? The transformer's causal attention mechanism means that each token attends to all preceding tokens but not to those that follow. The final subject token is the last position that "sees" the complete subject span before the relation and output positions begin. It becomes the natural aggregation point for information about the subject entity. The MLP layers at this position appear to serve as a content-addressable memory: given the accumulated subject representation, the MLP retrieves the associated facts encoded during pretraining.

This pattern is not universal across all types of knowledge. Facts that involve procedural or syntactic knowledge rather than entity-specific recall sometimes show more distributed effect patterns, with multiple positions and layers contributing roughly equally. For multi-hop reasoning, such as "What is the capital of the country where the Eiffel Tower is located?", the causal trace becomes considerably more complex, with multiple early sites corresponding to each retrieval step and potentially complex interactions between them.

By patching residual streams vs. MLP outputs vs. attention outputs separately, you can attribute effects to specific mechanisms within each layer. This is called component attribution and provides a finer-grained view of where computation happens, relative to where information happens to be present in the residual stream.

Consider the difference between patching a residual stream and patching just the MLP output at the same location. In a transformer with residual connections, each layer updates the residual stream through two sequential steps. First, the attention sublayer reads from the current residual stream and writes back an update:

Then the MLP sublayer processes that intermediate result and adds its own update:

where:

• $x^{l-1, p}$: the residual stream entering layer $l$ at position $p$
• $\text{Attn}^l(x^{l-1, p})$: the additive contribution from the multi-head attention sublayer at layer $l$
• $m^{l, p}$: the intermediate state after attention but before MLP at layer $l$, position $p$
• $\text{MLP}^l(m^{l, p})$: the additive contribution from the feed-forward sublayer at layer $l$, position $p$
• $x^{l, p}$: the full residual stream after all of layer $l$ at position $p$

Patching the full residual stream $x^{l, p}$ substitutes the combined output after both sublayers. Patching only the MLP output substitutes just $\text{MLP}^l(\cdot)$ while keeping the attention contribution from the corrupted run in place. Patching only the attention output substitutes $\text{Attn}^l(\cdot)$ while keeping the MLP contribution from the corrupted run. This decomposition reveals whether a layer's causal importance comes from its attention sublayer, which reads from other token positions, or its MLP sublayer, which processes the current position in isolation.

To see why this matters, consider what each sublayer does conceptually. The attention sublayer aggregates information from across the sequence: it reads from potentially all earlier positions and synthesizes a new representation by weighted combination. High attention patch effects indicate that the routing of information is causally important. The MLP sublayer operates position-wise: it applies a learned transformation to the current position's representation without cross-position communication. High MLP patch effects indicate that the local transformation at that position is causally important, which in the context of factual recall suggests that the MLP is doing the memory lookup.

When Meng et al. performed this decomposition, they found that for factual associations, MLP sublayers in the middle layers at the subject token position had much higher patch effects than attention sublayers at the same location. This asymmetry was one of the primary pieces of evidence for the "MLP as factual memory" interpretation, and it would have been invisible if the researchers had only examined residual stream patching without decomposing by component type.

Standard activation patching tells you which (layer, position) sites are causally important for a behavior. Path patching is a more precise variant that identifies specific computational paths between components, moving from "where" to "how information flows between where."

To understand why path patching is necessary, consider a simplified three-stage scenario: component A computes something, component B reads A's output and processes it, and component C reads B's output to produce the final answer. Standard patching might identify B as important because patching B's output changes the answer. But this doesn't tell you whether B is important because it reads from A specifically, or because it processes information that arrived through some other path, perhaps directly from the input embeddings or from a parallel attention head.

Path patching isolates specific edges in the computation graph. In the example, you would patch the output of A only as it feeds into B, while keeping everything else running from the corrupted input. If this recovers the clean behavior, you have demonstrated that the path A to B is causally important. If patching A's output globally has the same effect, then B isn't doing anything special with A's output: it's just passing A's output through unchanged. If patching A only into B is necessary, B is specifically processing A's output in an important way.

This distinction matters critically when you want to understand circuits: specific subnetworks of components that collaborate to implement a behavior. Path patching lets you verify whether a proposed circuit is complete (patching all edges in the circuit is sufficient to restore behavior) and minimal (removing any single edge breaks the restoration). These two properties, completeness and minimality, are the standard criteria for a circuit explanation to be considered a good account of the behavior.

Let $G$ be the computation graph of a transformer. Nodes are components (attention heads, MLP layers) and edges connect components whose outputs feed into others' inputs through the residual stream. For an edge from component $A$ to component $B$, path patching proceeds as follows:

1. Run a corrupted forward pass with all activations from $x_{\text{corrupt}}$.
2. At the point where $A$'s output would feed into $B$'s computation, substitute $A$'s output from the clean run instead.
3. Run the rest of the forward pass using corrupted values for all other inputs to $B$ and for all other edges in the graph.

The key difference from standard patching is that the clean activation of $A$ is substituted specifically for $B$'s input, not for any other downstream consumer of $A$'s output. Component $C$, which also reads from $A$, still sees the corrupted version of $A$'s output. This tests the specific edge $A \to B$ rather than $A$'s contribution globally.

Implementing path patching correctly requires more careful hook management than standard patching. You need hooks both at the output of $A$ (to capture the clean value) and at the input aggregation point of $B$ (to substitute only for $B$'s specific read of $A$). In transformer implementations where the residual stream is the shared communication channel, this means carefully tracking which additions to the residual stream came from which components, which requires either custom forward implementations or careful hook orchestration.

Path patching has been most extensively applied in circuit discovery work, where the goal is to identify the complete set of components and information flows that implement a specific capability. The Indirect Object Identification circuit study by Wang et al. used path patching to verify that the proposed circuit structure was both complete and minimal. The researchers identified candidate components through coarse activation patching, proposed a circuit hypothesis, then used path patching to test each edge in the proposed circuit individually.

The circuit for Indirect Object Identification involves several types of attention heads working in sequence. Duplicate token heads detect that a name appears twice in the input. Induction heads recognize the repeated-token pattern. S-inhibition heads receive information from the duplicate token detection and use it to suppress the output of name mover heads for the repeated name. Name mover heads then copy the remaining, non-suppressed name to the output. Each information flow, from duplicate token heads to S-inhibition heads, from S-inhibition heads to the suppression of name mover heads, was tested with path patching to verify that the specific edge (rather than only the destination component) is causally necessary.

To make these ideas concrete, consider the Indirect Object Identification (IOI) task studied by Wang et al. Given a sentence like "When Mary and John went to the store, John gave a bag to", the model should predict "Mary" because John is the repeated name and therefore the indirect object recipient should be the other person, Mary. This is a clean, well-defined task where the correct answer depends on detecting a specific structural pattern in the input: the repetition of one name.

The IOI task is particularly useful for circuit analysis for several reasons. The input is short and well-controlled. The correct answer is fully determined by the syntax of the input rather than world knowledge, so there is no ambiguity about what the model "should" do. The task requires multi-step reasoning: first identify all names, then detect which name is repeated, then select the non-repeated name. And the task can be varied along multiple dimensions (different name choices, different sentence templates) to test the generality of the discovered circuit.

Perhaps most importantly, the task is hard enough to be interesting but simple enough that a complete circuit explanation is tractable. Understanding how a model solves IOI provides a template for studying more complex behaviors, and the circuit discovery methodology developed for IOI has been applied to many other tasks since then.

Through causal tracing and path patching on GPT-2, Wang et al. identified a specific circuit responsible for IOI. The circuit involves multiple types of attention heads playing distinct roles:

• Duplicate token heads detect the repeated name (John) by attending strongly back to the previous occurrence of the same token. These heads run early in the network, typically in the first few layers.
• Induction heads detect the AB...A pattern, recognizing that a token appearing again has been seen before. Induction heads were previously studied by Olsson et al. and are known to form in two-layer circuits involving particular attention patterns.
• S-inhibition heads receive information from the duplicate token detection and suppress copying the subject name (John, the repeated one) to the output. These heads operate later in the network.
• Name mover heads identify name tokens and copy them to the output position. Without suppression, they would copy all names; with S-inhibition, only the non-repeated name (Mary) gets through.

The causal structure forms a chain. Duplicate token heads identify the repeated name and encode this information in their outputs. S-inhibition heads read from this output and use it to suppress name mover heads. Name mover heads then copy the remaining, non-suppressed name to the output. The model's prediction of "Mary" is the final result of this cascade.

Running activation patching on this task reveals that specific attention head outputs at specific token positions have outsized effects. The name mover heads at the last token position have very high patch effects because they directly control what name is copied to the output: these are the "last mile" components whose outputs most directly determine the prediction. S-inhibition heads at the position of the second occurrence of the subject name also have high effects, because they determine which name gets suppressed and therefore which name gets output.

By contrast, many other attention heads in the same layers show nearly zero patch effects, even though they process the same positions. This selectivity is what makes the circuit view convincing: it is not that all of layer 10 is important, but that specific heads in layer 10, those that implement the name moving computation, are important. This head-level specificity is only visible because activation patching was applied at head granularity rather than layer granularity.

The path patching results provide an even finer-grained picture. Patching the path from duplicate token heads to S-inhibition heads restores significant behavior, confirming that this edge is causally necessary. Patching the path from S-inhibition heads to name mover heads shows a strong suppression effect, confirming the inhibitory role. Each edge in the proposed circuit can be tested and quantified, building up a complete, verified account of how the model implements the behavior.

This level of mechanistic detail, identifying the layer, the head, the position, and the information communicated to each downstream head, is only possible because of the combination of causal tracing (for coarse attribution), component-level activation patching (for head-level attribution), and path patching (for edge-level verification).

Let's implement activation patching from scratch using a small transformer model. We'll work with GPT-2 and a factual recall task to demonstrate the core concepts.

We start by importing the necessary libraries and loading a pretrained model. GPT-2 is small enough to run on a laptop but large enough to exhibit interesting internal structure.

GPT-2 small has 12 layers, 12 attention heads per layer, and a hidden dimension of 768. This gives us a reasonable model to experiment with while keeping computation tractable.

PyTorch hooks allow us to intercept and record activations during a forward pass without modifying the model code. We register forward hooks on each transformer block to capture the residual stream after each layer. The hook fires after the block's computation and receives the output, which we clone and store.

The ActivationCache class manages all hooks in one place, which makes it easy to register them before a pass and remove them after. Forgetting to remove hooks is a common error: hooks persist across forward passes until explicitly removed, which would corrupt later experiments by caching stale values.

Now we define our clean and corrupted prompts and run forward passes to cache their activations. We use a simple factual recall task where the correct answer is unambiguous.

The clean run assigns high probability to "Paris" while the corrupted run naturally predicts "Berlin". This gives us a meaningful probability gap to work with in our patching experiments. The larger this gap, the more sensitive our normalized patch effect metric will be.

The core of activation patching is running the corrupted input while substituting clean activations at a specific layer. We use a hook that fires during the corrupted forward pass and replaces the output of a designated block with the corresponding cached clean activation.

The recovery percentage tells us how much each layer's activation contributes to the model knowing that "France" should predict "Paris". Layers with high recovery are causally important for this factual association. Note that patching early layers often shows little recovery because the information hasn't yet been fully assembled, while patching the very latest layers often shows near-complete recovery because we are providing activations very close to the output.

The standard way to present activation patching results is as a heatmap showing patch effects across all layers and token positions simultaneously. This gives an integrated view of the entire computation.

The heatmap reveals where in the network the factual association "France capital Paris" is causally encoded. Bright regions identify (layer, position) pairs where patching the residual stream from the clean run substantially restores the model's ability to predict "Paris". Look for an early bright cluster at the subject-related token positions in the middle layers, and a late bright cluster at the last token position in the final layers.

To better understand the layer-by-layer contribution, we can plot the maximum patch effect per layer. This marginalization over token positions shows which layers have the highest overall causal importance regardless of position.

Layers highlighted in red are the three most causally important for this factual association. The distribution of effects across layers reveals whether factual knowledge is concentrated in a narrow set of layers or distributed more broadly. A sharp distribution points to specific causal bottlenecks; a broad one suggests more distributed encoding.

To illustrate how component-type attribution works in practice, we can simulate the kind of decomposition that researchers perform when distinguishing MLP-driven effects from attention-driven effects. The simulation captures the pattern typically observed in factual association studies, where MLP and attention contributions peak at different layers.

The simulated plot illustrates why separating component types matters. If we only measured residual stream effects, we'd see a broadly raised curve and might conclude that many layers are roughly equally important. Separating MLP and attention effects reveals a more specific story: MLP layers in the middle of the network store and retrieve the factual association, while later attention layers are responsible for moving that information to the output position. The two mechanisms are complementary, not redundant.

Real interpretability work rarely relies on a single prompt pair. To build confidence that a finding reflects the model's general behavior rather than a quirk of one example, researchers aggregate patch effects across multiple semantically similar prompt pairs. Here we demonstrate this aggregation pattern with a small set of factual recall prompts.

Averaging across prompt pairs smooths out idiosyncratic patterns from individual examples and makes the underlying structure clearer. If the early-site MLP pattern at the subject position is real, it should survive averaging. If it was a fluke of one specific prompt, it will wash out. Reliable interpretability claims require this kind of multi-example validation.

Activation patching is a powerful technique, but several important limitations shape how you should interpret its results. Understanding these limitations is not a reason to avoid activation patching: it is essential for drawing the right conclusions from the experiments you run.

A high patch effect at position $(l, p)$ means that activation is sufficient to restore the behavior when transplanted from the clean run. It does not mean the activation is necessary. The network might have redundant pathways: multiple activations could each independently carry the relevant information. If you patch any one of them, the behavior recovers, but that doesn't mean the others are uninvolved.

This distinction matters for circuit discovery. If two attention heads both carry the same information and both feed into the final prediction, patching either one restores the behavior. But both heads are part of the circuit: removing either one would reduce performance. Standard activation patching only identifies sufficient sites, not all necessary sites.

Conversely, a low patch effect doesn't mean an activation is unimportant. An activation might be necessary but not sufficient: it carries some required information, but the patched run still fails because other required information from the clean run is absent. Recovering the behavior requires patching all required sites simultaneously, and patching one at a time misses the joint contribution. This is one reason why path patching, which can test multiple edges simultaneously in a proposed circuit, is more informative for circuit verification than single-site activation patching.

Patching activations from a different input creates an unusual combination: the patched run has clean activations at the patched site but corrupted activations everywhere else. This combination almost certainly never appeared in the training data. Transformers' attention mechanisms, which attend globally across positions, might behave in unexpected ways when one position has "clean" activations while surrounding positions carry corrupted values. The attention head doesn't know to treat that position differently; it processes the activation at face value, but the context it operates in is mismatched with what it saw during training.

The resulting activation pattern is off-distribution, and the model's behavior in this regime can be difficult to interpret. A high patch effect might simply indicate that the activation at the patched site has unusually high influence, not because it carries specific information, but because it is a coordinate in the embedding space that the model assigns high probability to a particular output regardless of context. Researchers have proposed variants like activation patching with mean ablation, which replaces activations with the average over a dataset rather than a specific clean example, to reduce this off-distribution concern. Another approach is to use corrupted inputs that are as similar as possible to the clean input to minimize distribution shift.

Causal tracing tells you where information is located at each layer and position, but not how that information is computed or represented. Knowing that MLP layer 8 at the subject token position is causally important for factual recall is a useful finding, but it doesn't tell you what the MLP is computing. Is it storing factual associations as key-value pairs in its weight matrices? Is it performing some form of associative lookup? Is the "factual information" a distributed representation that spans multiple layers and components?

Further mechanistic analysis is needed to answer these questions. The ROME paper followed up its causal tracing with a direct analysis of MLP weight matrices, interpreting the second-layer weight matrix as a value memory and showing that editing specific rows of this matrix could change the model's factual associations. This combination of causal tracing (to identify where to look) and weight analysis (to understand what is happening there) is more powerful than either technique alone.

Activation patching is best understood as a first step in a multi-step interpretability pipeline. It narrows the search space by identifying important locations, which then motivates targeted mechanistic analyses at those locations. The causal evidence from patching justifies the mechanistic investigation, preventing researchers from wasting effort on components that turn out to be causally irrelevant.

Results can be sensitive to the specific clean and corrupted prompts used. If the corrupted prompt produces a very similar baseline to the clean prompt (small $P_{\text{clean}} - P_{\text{orig}}$), the normalized patch effects are noisy and unreliable because we are dividing by a near-zero quantity. If the corrupted prompt is structurally very different (different length, different grammatical structure, different token overlap), the activations from the two runs come from incompatible computational contexts, and comparing them position by position may not be valid.

Best practice is to use corrupted prompts that are as syntactically similar as possible to the clean prompts while producing clearly different outputs. Noise corruption of embeddings handles this by keeping the token structure identical. Alternatively, using a minimal contrastive pair (changing only the subject entity) provides a clean comparison. Running experiments across multiple prompt pairs and checking that the pattern of effects is consistent across pairs provides confidence that the findings reflect model behavior rather than prompt-specific artifacts.

A full causal trace over all layers and token positions requires one patched forward pass for every (layer, position) pair. For a 12-layer model with an input of 10 tokens, that's 120 forward passes, which is fast on modern hardware. For large models with 100 or more layers, and for long-context tasks with hundreds or thousands of tokens, the total cost scales to tens of thousands of forward passes. At that scale, running a complete causal trace on a single prompt pair might take hours.

Several strategies reduce this cost. Batching multiple patches in a single forward pass by placing different patched examples in different batch items simultaneously can reduce the number of forward calls. Another approach is to focus the trace on a subset of layers identified by a coarser preliminary analysis, rather than sweeping all layers exhaustively. For long sequences, you can restrict the position sweep to positions that semantic analysis suggests are likely to be important. These approximations reduce cost at the expense of comprehensiveness.

A deeper limitation concerns the nature of neural network representations themselves. Much mechanistic interpretability research implicitly assumes that specific information is localized to specific activations: that "Paris-ness" can be attributed to a particular vector at a particular (layer, position). In reality, transformer representations exhibit superposition: many different features can be encoded simultaneously in the same high-dimensional vector by using different linear directions.

When you patch an entire residual stream vector, you are replacing all the features encoded in that vector simultaneously. If the vector encodes "this is France," "this is a capital city question," "this is a proper noun," and many other features, a high patch effect might reflect the combined contribution of all these features rather than the factual identity of the subject alone. Disentangling which features are responsible for the patch effect requires additional techniques such as linear representation probing, sparse autoencoders, or controlled ablations of specific directions within the activation space.

The combination of activation patching and sparse autoencoders has emerged as a particularly powerful workflow: use sparse autoencoders to decompose residual stream activations into interpretable monosemantic features, then use activation patching on individual features by patching just one feature direction rather than the entire vector. This approach addresses the superposition problem directly and provides much more fine-grained causal attribution than vector-level patching alone.

Activation patching does not stand alone. It fits into a broader ecosystem of interpretability tools, each contributing a different type of evidence about how neural networks work.

Probing and activation patching are complementary. Probing tells you what information is accessible in a representation: whether a linear classifier can decode entity type, sentiment, syntactic role, or other features from a given layer's activations. Activation patching tells you what information is causally used for a specific output. The combination of high probe accuracy and high patch effect provides the strongest evidence that a feature is both present and used. Probe accuracy without patch effect suggests the information is present but bypassed by the model's actual computation. Patch effect without probing accuracy suggests the information is carried in a nonlinear or distributed form that the probe cannot detect.

Attention weight visualization, one of the earliest interpretability methods for transformers, reveals which positions each attention head attends to. Activation patching complements this by testing whether those attention patterns are causally important. An attention head might attend strongly to a particular position, but if patching that head's output has no effect on the behavior, the attention pattern is decorative rather than functional. Conversely, a head that attends weakly but consistently to a specific position might have a high patch effect because even weak attention carries important information. Activation patching provides the causal grounding that attention visualization alone cannot.

Activation patching is one of the core tools in the mechanistic interpretability toolkit, alongside attention pattern analysis, weight analysis, circuit verification, and sparse autoencoders. The typical workflow uses activation patching early in the analysis pipeline to identify candidate components, then transitions to other tools to understand what those components compute. The circuit discovered through activation patching provides a hypothesis to be explained mechanistically, and the mechanistic explanation in turn provides a more principled understanding of why the circuit works.

Building on this foundation, the Logit Lens chapter will show how to interpret what information the residual stream is encoding at each layer by projecting it directly into vocabulary space. While activation patching asks "where does information flow", the logit lens asks "what is the network predicting at each layer along the way". Together they form a powerful pair for understanding the temporal dynamics of information processing in transformers.

Activation patching is a causal intervention technique that identifies which parts of a neural network are responsible for specific behaviors. By transplanting activations from a "clean" run into a "corrupted" run and observing whether the behavior recovers, you can attribute causal importance to individual (layer, position, component) combinations with precision that no correlation-based method can match.

The key ideas to carry forward are:

• Clean and corrupted runs: Define two inputs where one produces the target behavior and one doesn't. Cache activations from both runs. The gap between their outputs defines the scale of your intervention.
• Patch effect: The normalized recovery in target probability when substituting a clean activation into the corrupted run. Values near 1.0 indicate high causal importance; values near 0.0 indicate the site is causally irrelevant for this behavior.
• Causal tracing: Sweeping all (layer, position) combinations to produce a complete map of causal importance as a heatmap. The resulting picture reveals both early sites where information is first computed and late sites where it is propagated to the output.
• Component attribution: Patching residual streams, attention head outputs, or MLP outputs separately attributes importance to specific sublayers, distinguishing routing (attention) from retrieval (MLP).
• Path patching: Testing specific edges in the computation graph by patching the output of one component only as it feeds into another. This validates the edges of a proposed circuit, not just its nodes.
• Limitations: High patch effects indicate sufficiency, not necessity. Distribution shift from mismatched contexts is a real concern. Causal location identifies where to look, not what the computation is. Superposition means entire vector patches conflate multiple features. Reliable analyses aggregate over multiple prompt pairs and combine patching with complementary tools.

Activation patching opened the door to a new style of interpretability research: one that makes specific, falsifiable claims about which parts of a model compute which behaviors. Combined with the other tools in the mechanistic interpretability toolkit, it provides a rigorous foundation for understanding language models from the inside out.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about activation patching and causal tracing in neural networks.