Activation Patching › Worked Example: Indirect Object Identification

To make these ideas concrete, consider the Indirect Object Identification (IOI) task studied by Wang et al. Given a sentence like "When Mary and John went to the store, John gave a bag to", the model should predict "Mary" because John is the repeated name and therefore the indirect object recipient should be the other person, Mary. This is a clean, well-defined task where the correct answer depends on detecting a specific structural pattern in the input: the repetition of one name.

The IOI task is particularly useful for circuit analysis for several reasons. The input is short and well-controlled. The correct answer is fully determined by the syntax of the input rather than world knowledge, so there is no ambiguity about what the model "should" do. The task requires multi-step reasoning: first identify all names, then detect which name is repeated, then select the non-repeated name. And the task can be varied along multiple dimensions (different name choices, different sentence templates) to test the generality of the discovered circuit.

Perhaps most importantly, the task is hard enough to be interesting but simple enough that a complete circuit explanation is tractable. Understanding how a model solves IOI provides a template for studying more complex behaviors, and the circuit discovery methodology developed for IOI has been applied to many other tasks since then.

Through causal tracing and path patching on GPT-2, Wang et al. identified a specific circuit responsible for IOI. The circuit involves multiple types of attention heads playing distinct roles:

• Duplicate token heads detect the repeated name (John) by attending strongly back to the previous occurrence of the same token. These heads run early in the network, typically in the first few layers.
• Induction heads detect the AB...A pattern, recognizing that a token appearing again has been seen before. Induction heads were previously studied by Olsson et al. and are known to form in two-layer circuits involving particular attention patterns.
• S-inhibition heads receive information from the duplicate token detection and suppress copying the subject name (John, the repeated one) to the output. These heads operate later in the network.
• Name mover heads identify name tokens and copy them to the output position. Without suppression, they would copy all names; with S-inhibition, only the non-repeated name (Mary) gets through.

The causal structure forms a chain. Duplicate token heads identify the repeated name and encode this information in their outputs. S-inhibition heads read from this output and use it to suppress name mover heads. Name mover heads then copy the remaining, non-suppressed name to the output. The model's prediction of "Mary" is the final result of this cascade.

Running activation patching on this task reveals that specific attention head outputs at specific token positions have outsized effects. The name mover heads at the last token position have very high patch effects because they directly control what name is copied to the output: these are the "last mile" components whose outputs most directly determine the prediction. S-inhibition heads at the position of the second occurrence of the subject name also have high effects, because they determine which name gets suppressed and therefore which name gets output.

By contrast, many other attention heads in the same layers show nearly zero patch effects, even though they process the same positions. This selectivity is what makes the circuit view convincing: it is not that all of layer 10 is important, but that specific heads in layer 10, those that implement the name moving computation, are important. This head-level specificity is only visible because activation patching was applied at head granularity rather than layer granularity.

The path patching results provide an even finer-grained picture. Patching the path from duplicate token heads to S-inhibition heads restores significant behavior, confirming that this edge is causally necessary. Patching the path from S-inhibition heads to name mover heads shows a strong suppression effect, confirming the inhibitory role. Each edge in the proposed circuit can be tested and quantified, building up a complete, verified account of how the model implements the behavior.

This level of mechanistic detail, identifying the layer, the head, the position, and the information communicated to each downstream head, is only possible because of the combination of causal tracing (for coarse attribution), component-level activation patching (for head-level attribution), and path patching (for edge-level verification).