Mechanistic Interpretability › Circuits: The Unit of Analysis

The central concept in mechanistic interpretability is the circuit. A circuit is a subgraph of a neural network, consisting of specific neurons, attention heads, and the connections between them, that implements a particular computation. Just as a physical circuit board contains sub-circuits responsible for specific functions (the audio processing circuit, the memory controller circuit), a transformer model contains sub-circuits responsible for specific linguistic and reasoning functions.

The circuit viewpoint emerged from the observation that neural networks are not monolithic. Not every neuron participates equally in every computation. When a model predicts the next word in "The president signed the bill into", most of the network's capacity is irrelevant. A smaller subset of components, connected in specific ways, performs the semantic and syntactic work that leads to "law." Circuits are those relevant subsets.

This idea is not obvious in advance. One might expect neural network computations to be distributed across the network, with every parameter contributing a small amount to every prediction. Instead, researchers have repeatedly found that specific behaviors are implemented by surprisingly small subsets of parameters. The model devotes localized computational resources to localized tasks. This modularity is what makes mechanistic interpretability tractable, at least for well-defined behaviors in appropriately sized models.

To make this concrete, we need a precise vocabulary for describing neural network computations. A transformer consists of layers, each containing a multi-head attention module and an MLP (feed-forward) module. We can think of the outputs of each layer as "residual stream" vectors that accumulate information as the computation proceeds.

In this framing, a circuit consists of:

• Nodes: Attention heads, MLP neurons, or entire layers that contribute to the computation
• Edges: The information flow between nodes, mediated by the residual stream
• Weights: The specific weight matrices (query, key, value, output, MLP in/out) that implement the computation

A key insight from the mechanistic interpretability literature is that attention heads communicate with each other through the residual stream. Head A writes information to the residual stream, and head B in a later layer reads that information. This read-write structure, rather than direct head-to-head connections, is how multi-layer computations are organized. The residual stream functions as the bus over which all inter-component communication happens, and understanding which information travels on that bus at each position is central to circuit analysis.

To understand the read-write structure more concretely, consider what happens when attention head $h$ in layer $\ell$ processes a sequence. The head reads the current residual stream at each position by applying its query and key matrices. The resulting attention pattern determines which positions the head "looks at." Then, using the value and output matrices, the head writes a new vector to the residual stream at the current position, a vector that is a weighted combination of the information read from attended positions. Downstream heads in later layers read the residual stream again, now enriched with whatever the upstream head wrote. The circuit concept captures which of these reads and writes matter for a particular behavior.

Because all transformer components communicate through the residual stream via addition, the final representation $\mathbf{x}_L$ at layer $L$ for a given token can be decomposed as:

$$\mathbf{x}_L = \mathbf{x}_0 + \sum_{\ell=1}^{L} \left( \text{Attn}_\ell(\mathbf{x}_{\ell-1}) + \text{MLP}_\ell(\mathbf{x}_{\ell-1}) \right)$$

where:

• $\mathbf{x}_0$: the initial token embedding (input to the first layer)
• $\mathbf{x}_{\ell-1}$: the residual stream at the input to layer $\ell$
• $\text{Attn}_\ell$: the output of the attention module at layer $\ell$ (the sum over all heads)
• $\text{MLP}_\ell$: the output of the MLP module at layer $\ell$

This decomposition is powerful because it means each component's contribution to the final output can be studied in isolation. If you want to understand what a specific attention head contributes, you can look at the vector it adds to the residual stream and project it onto the output embedding space to see which tokens it promotes or suppresses. This projection approach makes it possible to read the functional meaning of a head's output in terms of vocabulary logits, the same space where the model's final prediction lives.

The decomposition also underlies a useful analysis technique called "logit attribution." Because the final logit for token $t$ is the inner product of the final residual stream with the unembedding row for $t$, and because the final residual stream is a sum of contributions, the logit can also be decomposed as a sum of contributions from each layer's attention and MLP output. This allows researchers to assign credit or blame for specific predictions to specific components.

The attention output for a single head can itself be decomposed. For head $h$ at layer $\ell$:

$$\text{Attn}_{\ell,h}(\mathbf{x}) = \mathbf{A}_{\ell,h} \cdot \mathbf{x} \cdot \mathbf{W}^V_{\ell,h} \cdot \mathbf{W}^O_{\ell,h}$$

where:

• $\mathbf{A}_{\ell,h}$: the attention weight matrix (which positions attend to which)
• $\mathbf{W}^V_{\ell,h}$: the value weight matrix
• $\mathbf{W}^O_{\ell,h}$: the output projection weight matrix

The product $\mathbf{W}^V_{\ell,h} \cdot \mathbf{W}^O_{\ell,h}$ is sometimes called the OV circuit (Output-Value circuit) of the head. It determines what information the head reads from the attended positions and how it transforms that information before writing to the residual stream.

Similarly, the product $\mathbf{W}^Q_{\ell,h} \cdot (\mathbf{W}^K_{\ell,h})^T$ is called the QK circuit (Query-Key circuit). It determines which positions attend to which based on their content in the residual stream.

Separating the OV and QK circuits is analytically useful because they perform distinct functions. The QK circuit answers "where should this head look?" while the OV circuit answers "given where it is looking, what does it write?" A head with a well-tuned QK circuit that attends to the right positions can still be useless if its OV circuit writes an uninformative vector. Conversely, a powerful OV circuit is meaningless without accurate attention targeting. Understanding a head requires understanding both circuits, and the product decomposition makes it possible to study each independently.

When researchers say two attention heads "compose," they mean the output of one head is read by the query, key, or value matrix of another. There are three types of composition:

• Q-composition: Head B reads Head A's output through its query matrix
• K-composition: Head B reads Head A's output through its key matrix
• V-composition: Head B reads Head A's output through its value matrix

Each type of composition creates a different functional relationship. V-composition is "information passing": Head A selects relevant content, and Head B further processes or re-routes it. Q-composition is "attention routing": Head A's output determines where Head B should look, allowing earlier layers to influence later attention patterns. K-composition is "relevance signaling": Head A's output determines which positions Head B considers relevant keys, allowing earlier computations to change the attractiveness of positions as attention targets.

Understanding which heads compose with each other is the key challenge in circuit analysis. A circuit for a specific behavior is exactly the set of heads and MLPs that participate in the computation, connected through the specific compositions that implement it. Two heads may both show high individual importance on a task (measured by ablation), yet fail to compose with each other. The information flowing from Head A might not reach Head B in a way that matters. Path patching (discussed below) is the technique that disambiguates this.

One useful quantitative measure of composition is the composition score. For Q-composition between Head A (layer $\ell_A$) and Head B (layer $\ell_B > \ell_A$), the score measures how much the output subspace of Head A overlaps with the query input subspace of Head B. Formally, if $W_A^O$ and $W_B^Q$ are the respective weight matrices, and if we compute the Frobenius norm of their product relative to the norms of the individual matrices, we get a measure of how strongly A's outputs are used by B's query computations. High composition scores indicate that the heads are functionally related, making them candidates for the same circuit.