Activation Patching › Summary

Activation patching is a causal intervention technique that identifies which parts of a neural network are responsible for specific behaviors. By transplanting activations from a "clean" run into a "corrupted" run and observing whether the behavior recovers, you can attribute causal importance to individual (layer, position, component) combinations with precision that no correlation-based method can match.

The key ideas to carry forward are:

• Clean and corrupted runs: Define two inputs where one produces the target behavior and one doesn't. Cache activations from both runs. The gap between their outputs defines the scale of your intervention.
• Patch effect: The normalized recovery in target probability when substituting a clean activation into the corrupted run. Values near 1.0 indicate high causal importance; values near 0.0 indicate the site is causally irrelevant for this behavior.
• Causal tracing: Sweeping all (layer, position) combinations to produce a complete map of causal importance as a heatmap. The resulting picture reveals both early sites where information is first computed and late sites where it is propagated to the output.
• Component attribution: Patching residual streams, attention head outputs, or MLP outputs separately attributes importance to specific sublayers, distinguishing routing (attention) from retrieval (MLP).
• Path patching: Testing specific edges in the computation graph by patching the output of one component only as it feeds into another. This validates the edges of a proposed circuit, not just its nodes.
• Limitations: High patch effects indicate sufficiency, not necessity. Distribution shift from mismatched contexts is a real concern. Causal location identifies where to look, not what the computation is. Superposition means entire vector patches conflate multiple features. Reliable analyses aggregate over multiple prompt pairs and combine patching with complementary tools.

Activation patching opened the door to a new style of interpretability research: one that makes specific, falsifiable claims about which parts of a model compute which behaviors. Combined with the other tools in the mechanistic interpretability toolkit, it provides a rigorous foundation for understanding language models from the inside out.