Feature Interpretation › Feature Circuits

Individual features do not operate in isolation. A model behavior is not produced by a single feature firing, but by a sequence of features activating across multiple layers, each influencing the next. Understanding a model behavior therefore requires understanding how features connect: which upstream features cause a given feature to activate, and which downstream features does it influence. The study of these connections is called feature circuit analysis.

A feature circuit is a subgraph of the transformer's computational graph in which the nodes are features (identified by SAE analysis) and the edges represent causal influences. An edge from feature $A$ to feature $B$ means that activating $A$ increases the probability that $B$ will activate, either directly (through the residual stream) or indirectly (through intermediate computations in attention or MLP layers).

The concept was formalized in the paper "Sparse Feature Circuits" by Marks et al. (2024), which showed that many model behaviors can be explained by small, compact circuits involving tens of features out of millions. For instance, the behavior "refuse to complete a toxic prompt" might be explained by a circuit in which a toxicity-detection feature in early layers activates a refusal-decision feature in a middle layer, which in turn modulates the output distribution in later layers to suppress harmful completions.

The appeal of the circuit framing is that it makes model behavior legible. Instead of asking "why did the model generate this output?", you can decompose the question into smaller, more tractable sub-questions: which features were active, which upstream features caused them, and which output logits did they influence? If the circuit is small, the answer is tractable. If the circuit is large, you have learned that the behavior is distributed and cannot be explained by a simple causal story, which is also informative.

This is why feature circuits are considered one of the most promising directions in mechanistic interpretability. They promise to reduce the analysis of arbitrarily complex model behaviors to a manageable graph problem, connecting the vocabulary of interpretable features to the downstream computation that produces model outputs.

The standard empirical method for identifying which features matter for a given behavior is activation patching, which we explored in an earlier chapter on interpretability foundations. In the context of feature circuits, activation patching works as follows. You define a "clean" input that produces the target behavior and a "corrupted" input that does not. You run both inputs through the model, storing activations at every SAE feature. Then you patch individual feature activations one at a time, replacing the clean value with the corrupted value, and measure how much the output changes.

Features for which patching causes a large change in output are causally relevant to the behavior. By composing patches at different layers, you can trace the causal chain from the input to the output, identifying the full circuit. The advantage of this approach over pure gradient attribution is that it is causal rather than correlational. A feature that is merely co-activated with the behavior (but not causally relevant) will produce little change when patched. A feature that is part of the circuit will produce a large change. This distinction matters enormously in practice: many features are activated by the same input distribution that triggers a behavior without being part of the mechanism that produces it.

Activation patching at the feature level is more computationally demanding than patching at the residual stream level, because you must run the SAE encoder and decoder for every position and every layer. In a large model with dozens of layers and millions of SAE features, this can require substantial compute. But the resulting circuits are far more interpretable because the nodes have meaningful labels, rather than being abstract residual stream vectors.

A practical refinement is hierarchical patching: rather than patching individual features, you first identify important layers (or attention heads) using residual-stream-level patching, and then apply feature-level patching only within those layers. This two-pass approach dramatically reduces the computational cost while preserving the interpretability of the final circuit.

For a feature at layer $l$, there are two symmetric questions about its causal role:

1. Upstream: Which features in earlier layers have the highest causal influence on this feature's activation? Answering this question traces the feature's causes.
2. Downstream: Which features in later layers does this feature causally influence? Answering this question traces the feature's effects.

These questions are addressed using gradient-based attribution. The upstream influence of feature $j$ at layer $l-1$ on feature $i$ at layer $l$ can be measured as:

$$\text{attr}(j \to i) = \frac{\partial a_i^{(l)}}{\partial a_j^{(l-1)}} \cdot a_j^{(l-1)}$$

where:

• $a_i^{(l)}$ is the activation of feature $i$ at layer $l$
• $a_j^{(l-1)}$ is the activation of feature $j$ at layer $l-1$
• $\frac{\partial a_i^{(l)}}{\partial a_j^{(l-1)}}$ is the gradient of feature $i$'s activation with respect to feature $j$'s activation, measuring how sensitive $i$ is to $j$
• The product of gradient and activation captures both the sensitivity and the actual magnitude of the source feature's contribution

This is a first-order linear attribution and is closely related to the "gradient times input" attribution method used in other interpretability contexts. For nonlinear interactions (for example, when an attention gate controls whether a feature's influence is transmitted), higher-order terms would be needed, but the first-order version already identifies the most important direct upstream contributors for most features.

To trace the full upstream circuit of feature $i$, you apply this attribution recursively: compute the upstream attributions for feature $i$, then compute the upstream attributions for the most important upstream features, and so on. You stop when you have traced back to the embedding layer or when the attribution scores become negligibly small. The resulting tree of attributions is the feature's upstream circuit.

The transformer's residual stream provides the natural framework for tracing feature circuits. Recall from our earlier discussion of transformer architecture that the residual stream at position $t$ and layer $l$ accumulates contributions from all previous attention heads and MLP layers:

$$\mathbf{x}^{(l)} = \mathbf{x}^{(0)} + \sum_{l'=1}^{l} \left(\mathbf{a}^{(l')} + \mathbf{m}^{(l')}\right)$$

where:

• $\mathbf{x}^{(0)}$ is the initial token embedding
• $\mathbf{a}^{(l')}$ is the summed output of all attention heads at layer $l'$
• $\mathbf{m}^{(l')}$ is the output of the MLP at layer $l'$

Each SAE applied to a particular layer $l$ operates on $\mathbf{x}^{(l)}$ and decomposes it into a sparse linear combination of feature vectors. When tracing circuits, we track how specific features in the decomposition at layer $l$ were caused by specific components (particular attention heads or MLP neurons) at earlier layers.

The key tool for this is the "direct path" decomposition: for a given feature $i$ at layer $l$, we ask which components of $\mathbf{x}^{(l)}$ most contributed to its activation. Because the residual stream is additive, we can decompose $\mathbf{x}^{(l)}$ into its constituent contributions from each upstream layer and component:

$$\mathbf{W}_{\text{enc}}^{(i)} \cdot \mathbf{x}^{(l)} = \mathbf{W}_{\text{enc}}^{(i)} \cdot \mathbf{x}^{(0)} + \sum_{l'=1}^{l} \left(\mathbf{W}_{\text{enc}}^{(i)} \cdot \mathbf{a}^{(l')} + \mathbf{W}_{\text{enc}}^{(i)} \cdot \mathbf{m}^{(l')}\right)$$

Each term in this sum is the attribution of a specific upstream component to feature $i$'s pre-activation. Components with large positive terms are the main contributors to the feature's activation; components with large negative terms suppress it.

Attention heads are a particularly important node type in feature circuits. An attention head reads from the residual stream at query and key positions, computes attention weights over the sequence, and writes a value vector back to the residual stream at each position. If we have SAE features at both the input and output of an attention layer, we can measure how strongly each head's output contributes to each output feature.

Let $\mathbf{v}_h$ be the value vector written by attention head $h$ at token position $t$. The contribution of head $h$ to output feature $i$ is:

$$c_{h \to i} = \mathbf{W}_{\text{enc}}^{(i)} \cdot \mathbf{v}_h$$

where $\mathbf{W}_{\text{enc}}^{(i)}$ is the encoder weight vector for feature $i$. This is a direct measure of whether head $h$'s output aligns with the direction of feature $i$ in residual stream space. A large positive value means head $h$ is writing in the direction that activates feature $i$; a large negative value means it is suppressing the feature.

By computing this for all heads and all features, you get an attribution matrix. Heads that strongly write in the direction of a feature are part of that feature's upstream circuit. This connects the SAE feature analysis directly to the attention pattern analysis: you can ask which attention head is important and which source positions that head is attending to when it writes the feature-activating information. The full causal chain then runs from a source token, through attention weights that select that token, through a value computation that extracts information from the source token's representation, to the residual stream update, to the SAE feature activation, to the downstream features and ultimately to the model's output logits.