Activation Patching › The Patching Methodology

Activation patching works by carefully comparing what happens when a model processes two different inputs: a "clean" input that produces the behavior you want to understand, and a "corrupted" input that disrupts that behavior. The methodology is deceptively simple to describe but requires careful thought to execute and interpret correctly.

The procedure begins by defining two inputs that differ in a meaningful way. In the classic factual association setting from the ROME paper, the clean prompt might be "The Eiffel Tower is located in" and the corrupted prompt might be "The Colosseum is located in". The model answers the clean prompt with "Paris" and the corrupted prompt with "Rome". Now you want to know: which activations during the clean run are responsible for the model outputting "Paris"?

The patching experiment works as follows. You run the model on both inputs and cache all intermediate activations. Then you run the model again on the corrupted input, but at a specific position and layer, you replace the activation from the corrupted run with the corresponding activation from the clean run. You observe whether the model's output shifts back toward "Paris". The key intuition is: if swapping that single activation changes the output toward the clean prediction, then that activation was carrying information that the model needs for the clean behavior.

More precisely, the three runs are:

• Clean run: Process input $x_{\text{clean}}$, saving all activations $\{a^{l, p}_{\text{clean}}\}$ for every layer $l$ and token position $p$.
• Corrupted run: Process input $x_{\text{corrupt}}$, saving all activations $\{a^{l, p}_{\text{corrupt}}\}$.
• Patched run: Process $x_{\text{corrupt}}$ but at a specific (layer $l^*$, position $p^*$), replace $a^{l^*, p^*}_{\text{corrupt}}$ with $a^{l^*, p^*}_{\text{clean}}$. All other activations are computed normally from the corrupted input.

If the patched run's output recovers toward the clean output, the patched activation was causally important. If the output doesn't change, that activation carries no information the model is using for this specific behavior.

To quantify how much a patch recovers the clean behavior, researchers typically track the probability or log-probability of the clean answer token. Let $P_{\text{orig}}$ be the model's probability of the correct answer under the corrupted run (the baseline without any patching), and let $P_{\text{clean}}$ be the probability under the unmodified clean run. For a patched run where the activation at position $(l^*, p^*)$ has been replaced, the probability of the target token is $P_{\text{patched}}$.

The normalized patch effect is defined as:

$$\text{Effect}(l^*, p^*) = \frac{P_{\text{patched}} - P_{\text{orig}}}{P_{\text{clean}} - P_{\text{orig}}}$$

where:

• $l^*$: the layer index being patched
• $p^*$: the token position index being patched
• $P_{\text{orig}}$: probability of the clean answer under the unpatched corrupted run, serving as the zero-recovery baseline
• $P_{\text{clean}}$: probability of the clean answer under the clean run, serving as the full-recovery ceiling
• $P_{\text{patched}}$: probability of the clean answer after transplanting the activation at $(l^*, p^*)$ from the clean run into the corrupted run

A value of 1.0 means the patch fully restores the clean behavior: transplanting just this one activation was sufficient to recover the correct answer. A value of 0.0 means the patch has no effect at all. Values between 0 and 1 indicate partial recovery, where the patched site contributes some information but not all that is needed to fully recover the behavior.

This normalization is essential for making results comparable across different prompts and models. Without it, a small absolute probability gap (say, 0.03 to 0.05) and a large one (say, 0.10 to 0.85) would be on incomparable scales. The normalized effect always lies in $[0, 1]$ when the patch cannot harm the corrupted run beyond its baseline, though values slightly outside this range can occur due to nonlinearities in the model.

Probability is preferred because probability is preferred over logit difference or some other metric. Probability is bounded and interpretable, and it directly reflects what matters in practice: the model's confidence in the correct answer. Log-probability is sometimes used instead because it has better numerical properties when probabilities are very small or very large, but normalized probability works well for most analyses.

The granularity of what you patch matters enormously, because each choice asks a fundamentally different question about the model's computation. Common choices include:

• Residual stream vectors: The main hidden state at each position after each layer. Patching a residual stream vector at position $p$ and layer $l$ tests whether the total information at that point in the computation is causally important for the behavior.
• Attention head outputs: The output of a specific attention head, which is a vector in the residual stream dimension that gets added to the stream. Patching an individual head output isolates that head's contribution, asking whether this specific head's reading-and-writing operation is causally involved.
• MLP layer outputs: The output of the feed-forward sublayer at a specific position. Patching this substitutes the MLP's transformation while leaving the attention contribution from the corrupted run in place.
• Attention patterns: The attention weight matrix itself, independent of the value vectors. Patching this tests whether the routing of information is causally important, separate from the content of what is being routed.
• Key, query, or value projections: Even finer-grained components within a single attention head, allowing you to ask whether a head's effect comes from what it attends to, what information it writes, or the combination.

Each choice asks a different question. Patching residual streams gives a coarse view of which layer and position matter. Patching individual heads or MLP layers provides finer attribution. Patching attention patterns specifically tests the routing computation. The choice should match the question you are trying to answer, and in practice researchers often use multiple granularities in sequence: start coarse with residual stream patching to identify important (layer, position) pairs, then drill down with component-level patching to understand the mechanism.