Activation Patching › Limitations and Practical Considerations

Activation patching is a powerful technique, but several important limitations shape how you should interpret its results. Understanding these limitations is not a reason to avoid activation patching: it is essential for drawing the right conclusions from the experiments you run.

A high patch effect at position $(l, p)$ means that activation is sufficient to restore the behavior when transplanted from the clean run. It does not mean the activation is necessary. The network might have redundant pathways: multiple activations could each independently carry the relevant information. If you patch any one of them, the behavior recovers, but that doesn't mean the others are uninvolved.

This distinction matters for circuit discovery. If two attention heads both carry the same information and both feed into the final prediction, patching either one restores the behavior. But both heads are part of the circuit: removing either one would reduce performance. Standard activation patching only identifies sufficient sites, not all necessary sites.

Conversely, a low patch effect doesn't mean an activation is unimportant. An activation might be necessary but not sufficient: it carries some required information, but the patched run still fails because other required information from the clean run is absent. Recovering the behavior requires patching all required sites simultaneously, and patching one at a time misses the joint contribution. This is one reason why path patching, which can test multiple edges simultaneously in a proposed circuit, is more informative for circuit verification than single-site activation patching.

Patching activations from a different input creates an unusual combination: the patched run has clean activations at the patched site but corrupted activations everywhere else. This combination almost certainly never appeared in the training data. Transformers' attention mechanisms, which attend globally across positions, might behave in unexpected ways when one position has "clean" activations while surrounding positions carry corrupted values. The attention head doesn't know to treat that position differently; it processes the activation at face value, but the context it operates in is mismatched with what it saw during training.

The resulting activation pattern is off-distribution, and the model's behavior in this regime can be difficult to interpret. A high patch effect might simply indicate that the activation at the patched site has unusually high influence, not because it carries specific information, but because it is a coordinate in the embedding space that the model assigns high probability to a particular output regardless of context. Researchers have proposed variants like activation patching with mean ablation, which replaces activations with the average over a dataset rather than a specific clean example, to reduce this off-distribution concern. Another approach is to use corrupted inputs that are as similar as possible to the clean input to minimize distribution shift.

Causal tracing tells you where information is located at each layer and position, but not how that information is computed or represented. Knowing that MLP layer 8 at the subject token position is causally important for factual recall is a useful finding, but it doesn't tell you what the MLP is computing. Is it storing factual associations as key-value pairs in its weight matrices? Is it performing some form of associative lookup? Is the "factual information" a distributed representation that spans multiple layers and components?

Further mechanistic analysis is needed to answer these questions. The ROME paper followed up its causal tracing with a direct analysis of MLP weight matrices, interpreting the second-layer weight matrix as a value memory and showing that editing specific rows of this matrix could change the model's factual associations. This combination of causal tracing (to identify where to look) and weight analysis (to understand what is happening there) is more powerful than either technique alone.

Activation patching is best understood as a first step in a multi-step interpretability pipeline. It narrows the search space by identifying important locations, which then motivates targeted mechanistic analyses at those locations. The causal evidence from patching justifies the mechanistic investigation, preventing researchers from wasting effort on components that turn out to be causally irrelevant.

Results can be sensitive to the specific clean and corrupted prompts used. If the corrupted prompt produces a very similar baseline to the clean prompt (small $P_{\text{clean}} - P_{\text{orig}}$), the normalized patch effects are noisy and unreliable because we are dividing by a near-zero quantity. If the corrupted prompt is structurally very different (different length, different grammatical structure, different token overlap), the activations from the two runs come from incompatible computational contexts, and comparing them position by position may not be valid.

Best practice is to use corrupted prompts that are as syntactically similar as possible to the clean prompts while producing clearly different outputs. Noise corruption of embeddings handles this by keeping the token structure identical. Alternatively, using a minimal contrastive pair (changing only the subject entity) provides a clean comparison. Running experiments across multiple prompt pairs and checking that the pattern of effects is consistent across pairs provides confidence that the findings reflect model behavior rather than prompt-specific artifacts.

A full causal trace over all layers and token positions requires one patched forward pass for every (layer, position) pair. For a 12-layer model with an input of 10 tokens, that's 120 forward passes, which is fast on modern hardware. For large models with 100 or more layers, and for long-context tasks with hundreds or thousands of tokens, the total cost scales to tens of thousands of forward passes. At that scale, running a complete causal trace on a single prompt pair might take hours.

Several strategies reduce this cost. Batching multiple patches in a single forward pass by placing different patched examples in different batch items simultaneously can reduce the number of forward calls. Another approach is to focus the trace on a subset of layers identified by a coarser preliminary analysis, rather than sweeping all layers exhaustively. For long sequences, you can restrict the position sweep to positions that semantic analysis suggests are likely to be important. These approximations reduce cost at the expense of comprehensiveness.

A deeper limitation concerns the nature of neural network representations themselves. Much mechanistic interpretability research implicitly assumes that specific information is localized to specific activations: that "Paris-ness" can be attributed to a particular vector at a particular (layer, position). In reality, transformer representations exhibit superposition: many different features can be encoded simultaneously in the same high-dimensional vector by using different linear directions.

When you patch an entire residual stream vector, you are replacing all the features encoded in that vector simultaneously. If the vector encodes "this is France," "this is a capital city question," "this is a proper noun," and many other features, a high patch effect might reflect the combined contribution of all these features rather than the factual identity of the subject alone. Disentangling which features are responsible for the patch effect requires additional techniques such as linear representation probing, sparse autoencoders, or controlled ablations of specific directions within the activation space.

The combination of activation patching and sparse autoencoders has emerged as a particularly powerful workflow: use sparse autoencoders to decompose residual stream activations into interpretable monosemantic features, then use activation patching on individual features by patching just one feature direction rather than the entire vector. This approach addresses the superposition problem directly and provides much more fine-grained causal attribution than vector-level patching alone.