Activation Patching › Locating Information

Activation patching is particularly powerful for locating where specific types of information are encoded in a model, and the granularity of this localization has proven useful for guiding subsequent mechanistic work.

In factual recall tasks, a prompt like "The Eiffel Tower is located in" contains at least two semantic components worth distinguishing: the subject ("The Eiffel Tower") and the relation ("is located in"). The object ("Paris") is what the model must retrieve. Causal tracing can reveal which component's processing is most critical for retrieval, which in turn informs theories of how factual knowledge is organized.

The intuition behind this decomposition is that different components of a fact may be processed at different locations and times in the transformer. The subject identifies which entity we are asking about. The relation specifies what type of information to retrieve about that entity. A model that has learned factual associations might process the subject to construct an entity representation, then use the relation to select which fact to retrieve, and finally generate the object as output.

Empirically, researchers have found that for factual associations:

• Subject tokens, especially the last subject token, carry the most critical information in early-to-middle layers, particularly in MLP sublayers. The last subject token aggregates information about the entire subject through causal attention and is the natural site for entity-level processing.
• Relation tokens have smaller effects in most cases, though they can be important for specific relation types where the relation itself disambiguates among multiple plausible answers.
• The final token position sees high effects in the last few layers, showing the model assembling the answer right before output, which is the late-site propagation described above.

Why does the last subject token carry so much weight? The transformer's causal attention mechanism means that each token attends to all preceding tokens but not to those that follow. The final subject token is the last position that "sees" the complete subject span before the relation and output positions begin. It becomes the natural aggregation point for information about the subject entity. The MLP layers at this position appear to serve as a content-addressable memory: given the accumulated subject representation, the MLP retrieves the associated facts encoded during pretraining.

This pattern is not universal across all types of knowledge. Facts that involve procedural or syntactic knowledge rather than entity-specific recall sometimes show more distributed effect patterns, with multiple positions and layers contributing roughly equally. For multi-hop reasoning, such as "What is the capital of the country where the Eiffel Tower is located?", the causal trace becomes considerably more complex, with multiple early sites corresponding to each retrieval step and potentially complex interactions between them.

By patching residual streams vs. MLP outputs vs. attention outputs separately, you can attribute effects to specific mechanisms within each layer. This is called component attribution and provides a finer-grained view of where computation happens, relative to where information happens to be present in the residual stream.

Consider the difference between patching a residual stream and patching just the MLP output at the same location. In a transformer with residual connections, each layer updates the residual stream through two sequential steps. First, the attention sublayer reads from the current residual stream and writes back an update:

$$m^{l, p} = x^{l-1, p} + \text{Attn}^l(x^{l-1, p})$$

Then the MLP sublayer processes that intermediate result and adds its own update:

$$x^{l, p} = m^{l, p} + \text{MLP}^l(m^{l, p})$$

where:

• $x^{l-1, p}$: the residual stream entering layer $l$ at position $p$
• $\text{Attn}^l(x^{l-1, p})$: the additive contribution from the multi-head attention sublayer at layer $l$
• $m^{l, p}$: the intermediate state after attention but before MLP at layer $l$, position $p$
• $\text{MLP}^l(m^{l, p})$: the additive contribution from the feed-forward sublayer at layer $l$, position $p$
• $x^{l, p}$: the full residual stream after all of layer $l$ at position $p$

Patching the full residual stream $x^{l, p}$ substitutes the combined output after both sublayers. Patching only the MLP output substitutes just $\text{MLP}^l(\cdot)$ while keeping the attention contribution from the corrupted run in place. Patching only the attention output substitutes $\text{Attn}^l(\cdot)$ while keeping the MLP contribution from the corrupted run. This decomposition reveals whether a layer's causal importance comes from its attention sublayer, which reads from other token positions, or its MLP sublayer, which processes the current position in isolation.

To see why this matters, consider what each sublayer does conceptually. The attention sublayer aggregates information from across the sequence: it reads from potentially all earlier positions and synthesizes a new representation by weighted combination. High attention patch effects indicate that the routing of information is causally important. The MLP sublayer operates position-wise: it applies a learned transformation to the current position's representation without cross-position communication. High MLP patch effects indicate that the local transformation at that position is causally important, which in the context of factual recall suggests that the MLP is doing the memory lookup.

When Meng et al. performed this decomposition, they found that for factual associations, MLP sublayers in the middle layers at the subject token position had much higher patch effects than attention sublayers at the same location. This asymmetry was one of the primary pieces of evidence for the "MLP as factual memory" interpretation, and it would have been invisible if the researchers had only examined residual stream patching without decomposing by component type.