Feature Interpretation: SAE Features, Naming, and Circuits

In the previous chapter, we learned how Sparse Autoencoders (SAEs) decompose the internal representations of language models into a large dictionary of sparse features. Training an SAE gives you a set of feature directions in activation space, each capturing some latent concept the model has encoded. But having thousands of abstract feature vectors is only the beginning. The central challenge of feature interpretation is figuring out what each of those features means: which concepts do they track, how strongly do they fire across different contexts, and how do they connect to one another in the broader circuitry of the network?

This chapter covers the full workflow for making sense of SAE features. We start with the basics of activation patterns: how features fire on example inputs, and what statistical signatures distinguish sharp, monosemantic features from vague or polysemantic ones. We then move to feature naming, both manual and automated, exploring how large language models can be used as automated interpreters. From there, we study the geometry of feature space, asking how features cluster, point in related directions, and form linear subspaces. We cover causal validation through feature steering and ablation, which let you test whether a feature merely correlates with a behavior or drives it. Finally, we tackle feature circuits: the downstream and upstream connections that link one feature to others, revealing functional pathways inside the transformer. The next chapter on Mechanistic Interpretability will build on these circuits and show how entire algorithms can be reverse-engineered from the weights.

The most direct way to understand a feature is to look at which inputs activate it. Before you can name a feature or understand its role in a circuit, you need to observe the raw activation signal: when does it fire, how strongly, and over what range of contexts?

A feature direction $\mathbf{f} \in \mathbb{R}^d$ fires on a token $t$ in context $C$ with activation:

where:

• $\mathbf{W}_{\text{enc}}^{(i)}$ is the $i$-th row of the SAE encoder weight matrix, representing the feature direction
• $\mathbf{x}$ is the residual stream activation at the position of token $t$
• $b_i$ is the learned bias (the pre-activation threshold) for feature $i$
• $a$ is the activation value, zero when the feature does not fire and positive when it does

The ReLU function serves two purposes here. First, it enforces non-negativity: activations represent the degree to which the feature is present, and a negative projection does not represent "negative presence" in a useful sense. Second, the bias term $b_i$ learned during SAE training acts as a threshold below which the feature is considered absent. The SAE training process learns this threshold jointly with the encoder and decoder weights, so the threshold is calibrated to keep the representation sparse while minimizing reconstruction error.

To study activation patterns empirically, you collect the top-$k$ activating examples from a large corpus for each feature. The standard procedure is to run the model over a large dataset, record the activation value $a$ for each token position, and then sort by $a$ to find the examples where the feature fires most strongly. For practical feature analysis, this typically requires running inference over millions to billions of tokens, which is computationally demanding but feasible on modern hardware for models up to a few billion parameters.

The "max-activating examples" approach, popularized by Anthropic and used extensively in the paper "Towards Monosemanticity," shows the most and least activating contexts for each feature. The idea is simple: if a feature is tracking a coherent concept, its top-$k$ activating tokens and surrounding context should cluster around a recognizable theme.

For example, a feature that fires highly on tokens like "Paris", "London", "Berlin" when used as subjects of verbs like "is the capital of" is revealing information about capital-city relationships. A feature that fires on words like "however", "nevertheless", "despite" is tracking concessive discourse connectives. A feature that fires on tokens in Python docstrings but not in prose is tracking a structural property of source code. These patterns emerge directly from looking at examples, without any supervision or pre-specified concept taxonomy.

The key insight is that you can observe the distribution of activating examples and use it to formulate a hypothesis about the feature's function. This hypothesis can then be tested by finding examples that should activate it (if it tracks concept X, does it fire on all instances of X?) and examples that should not (is there cross-activation on unrelated concepts?). The process is fundamentally empirical and iterative: your first hypothesis will often be wrong or incomplete, and revising it requires going back to more examples.

In practice, researchers display max-activating examples with the activating token highlighted and several tokens of surrounding context shown. The activation value is color-coded by intensity: tokens that cause stronger activations are shown with deeper color. This visualization immediately reveals whether the feature is tracking the highlighted token itself, a property of its grammatical context, or something in the broader paragraph context. A feature that fires when "bank" is the highlighted token but only in certain sentence contexts is doing something more sophisticated than simply recognizing the word form.

Features in a well-trained SAE have a characteristic activation histogram: a large mass of zeros (the sparse regime, where the feature does not fire) and a long tail of positive activations (the active regime). The shape of this histogram is diagnostic. A feature that fires broadly and weakly may be polysemantic or a catch-all, serving as a residual absorber for variance that the SAE cannot explain with sharper features. A feature that fires rarely but intensely is likely tracking a specific, rare concept, such as a technical term, a named entity, or a particular grammatical construction that appears infrequently in the training corpus.

You can quantify this with a few statistics:

• Activation frequency: the fraction of tokens in the corpus for which $a > 0$. Well-interpreted features often have frequencies ranging from $10^{-4}$ to $10^{-1}$.
• Mean active activation: the average value of $a$ conditioned on $a > 0$. A high mean active activation relative to the threshold indicates a feature that fires strongly when it fires at all.
• Log activation rank: the rank of a feature by frequency. Frequency distributions across a trained SAE are approximately log-uniform, similar to Zipf's law in natural language. This is not a coincidence: the training corpus itself follows approximately Zipfian statistics, so features that track concept frequency inherit this distribution.

Features near the high-frequency end of the spectrum deserve special attention. Very frequent features often turn out to track basic syntactic or structural properties: "token is part of a number", "token appears at the end of a sentence", "token is in all-caps". These are real features of language, but they are not particularly informative about the model's higher-level knowledge. Very rare features may track niche concepts, or they may be "dead features" that never fire due to poor initialization during SAE training.

Dead features deserve special mention. In a large SAE dictionary, a substantial fraction of features may have activation frequency effectively zero over any reasonable-sized corpus. This happens because SAE training with an $L_1$ sparsity penalty can collapse certain feature directions: the optimizer finds it easier to set a feature's activation to zero for all inputs than to learn a specialized direction that would fire infrequently. Detecting dead features is important for quality control, as they represent wasted capacity in the dictionary. Heuristically, features with activation frequency below $10^{-6}$ over a corpus of one million tokens should be treated as dead.

An important distinction is the difference between a feature's token-level activation and its context-level meaning. SAEs typically operate on residual stream vectors at individual token positions, so the activation $a$ is computed separately for each token. But the meaning of an activation often depends heavily on the surrounding context.

For instance, a feature might fire strongly on the token "bank" only in the context of financial sentences ("I deposited money at the bank"), not when "bank" refers to a river bank or a data bank. This context-sensitivity is a sign that the model has already resolved polysemy before the SAE sees the activation: the residual stream carries the contextual interpretation of the token, not just its raw identity. The attention heads in earlier layers have already integrated the surrounding tokens to produce a context-sensitive representation, and the SAE operates on that already-contextualized vector.

This distinction is important for interpretation in two ways. First, when studying max-activating examples, you should always display enough surrounding context to understand what the model was computing, in addition to the highlighted token. A feature described as "the token 'bank'" is almost certainly wrong: it is more accurately described as "financial institutions in context." Second, it means that features at later layers tend to be more abstract and context-sensitive than features at earlier layers, which are closer to the raw token embeddings and thus more often track surface-level properties like token identity, capitalization, or word length.

The contrast between early-layer and late-layer features is one of the more striking empirical findings in SAE research. Early-layer features tend to look like enhanced versions of word embeddings: they capture lexical meaning, part of speech, and basic morphological properties. Late-layer features often capture more abstract relational properties: whether this token is part of an argument structure, whether the current text is describing a causal relationship, whether the model is in the middle of a list or a narrative.

Identifying what a feature represents requires assigning it a human-readable label. Feature naming is the bridge between raw activation patterns and interpretable descriptions, and it is both more art than science and more science than art: there are principled methods for evaluating descriptions, but the initial hypothesis generation is largely a creative act.

In the original work on sparse autoencoders applied to small transformer models (including Anthropic's work on toy models of superposition and the subsequent "Towards Monosemanticity" paper), researchers assigned names to features manually by inspecting the top-activating examples. This process is labor-intensive but highly reliable when done carefully.

The workflow for manual interpretation involves several steps. First, for each feature, you retrieve the top 20 to 50 examples (or tokens) by activation strength. You read these examples carefully, giving special attention to both the token highlighted with maximum activation and the surrounding context. Second, you identify a common theme across the examples. This requires relevant linguistic and domain knowledge: a researcher without knowledge of Python syntax will struggle to label features that fire on docstrings or type annotations. Third, you propose a short label that is precise and falsifiable, such as "Hebrew text", "Python variable names", or "negative sentiment adjectives." Finally, you validate the label by checking for false positives (do any top-activating examples not match the description?) and false negatives (do known instances of the concept fail to activate the feature?).

Manual naming is valuable because experienced researchers notice nuances that automated systems miss. They might notice that a feature fires on both formal titles ("Dr.", "Prof.") and on abbreviations in scientific notation ("mL", "cm"), which would lead to a more careful decomposition rather than a single imprecise label. They might notice that a feature fires more strongly on the first occurrence of a named entity in a passage than on subsequent mentions, revealing something about the model's coreference tracking. These observations require the interpretability researcher to bring substantial world knowledge and linguistic intuition to the task, which cannot be fully automated.

The main drawback is scale. Modern SAEs trained on large language models have millions of features. Inspecting them individually at the level of quality described above is not feasible. Researchers at major labs have noted that careful manual labeling of a single feature can take several minutes, which means that a million-feature SAE would require millions of person-hours to label manually. This scaling gap is the primary motivation for automated interpretation pipelines.

To scale feature naming, researchers use language models themselves as automated interpreters. The general pipeline converts the problem of feature labeling into a language modeling task: given the top-$k$ activating examples for a feature, ask a language model to describe what they have in common.

The pipeline has three stages:

1. Collection: For each feature, retrieve the top-$k$ activating examples from the corpus, preserving both the highlighted token and its context window. The number of examples $k$ is typically between 10 and 30: too few and the description generator has insufficient evidence; too many and the prompt becomes unwieldy and expensive.

2. Generation: Format the examples as a prompt that asks a language model to describe the common pattern. A typical prompt might present the examples with activation values and ask: "Based on these contexts, what pattern does this feature appear to track? Provide a concise description in one or two sentences."

3. Calibration: Score the proposed description by testing it as a predictor on held-out examples. The scoring step is critical and is what distinguishes a rigorous automated pipeline from simple summarization.

The calibration step deserves elaboration. A description that says "mentions emotions" is nearly vacuous: it matches a broad swath of text and does not make useful predictions about which specific contexts will cause the feature to fire. The ideal description makes precise, falsifiable predictions. To measure this, the calibration stage uses a simulator: another language model is given the proposed description and asked, for each token in a new passage, whether the feature should be active at that position. The simulated activations are then compared against the actual feature activations, and the correlation is used as the calibration score.

Anthropic's automated interpretability pipeline, used in their feature analysis of Claude models and in the open-source Neuronpedia tool, formalizes this into three steps: simulation, description, and calibration. The simulator takes a proposed description and predicts which tokens in a new context should cause the feature to fire; the calibration metric measures the Pearson correlation between the simulator's predicted activations and the actual activations from the SAE.

The automated interpretation pipeline reveals a striking finding: a large fraction of SAE features trained on frontier language models have interpretable, human-understandable descriptions with autoint scores well above chance. In Anthropic's analysis of a one-billion-parameter model, a majority of features were judged as cleanly monosemantic, with a smaller fraction covering polysemantic or unclear behaviors. This result supports the core theoretical prediction of the superposition hypothesis: that SAE decomposition should recover features that are more semantically coherent than raw neurons.

An important practical observation is that the "vocabulary" of features changes qualitatively across layers of the transformer. When you apply SAEs to multiple layers of the same model and analyze the resulting features, you see a rough semantic hierarchy emerge.

In the first few layers, features tend to be near-syntactic: they track properties like part of speech, capitalization, whether a token is numeric, whether it appears in a URL, whether it is a punctuation mark. These features are relatively easy to interpret and tend to have high autoint scores, because the mapping from token identity to concept is fairly direct.

In the middle layers, features shift toward local semantic properties: named entity types (person, organization, location), sentiment polarity, domain indicators (medical, legal, scientific), discourse structure markers. These features are somewhat harder to interpret because they depend on context rather than the token alone, but they are still often cleanly monosemantic.

In the final layers, features tend to be abstract and relational: they may capture properties like "this is the conclusion of an argument", "this is a counterfactual statement", "this token is part of a reasoning chain." These features are the hardest to interpret and often have lower autoint scores, showing the difficulty of expressing abstract relational properties in natural language.

This layer-wise progression is an empirical observation and an important guide for practitioners. If you are trying to understand a model's behavior on a specific task, the layer at which you apply the SAE matters enormously. Features at early layers will tell you about input preprocessing and token-level properties. Features at late layers will tell you about the model's high-level reasoning and output planning. For most interpretability questions about model behavior, the most informative features tend to be in the middle to late layers.

A core goal of feature interpretation is to measure the degree of monosemanticity: how well a single feature corresponds to a single concept. The theory of superposition, discussed in the previous chapter on SAEs, predicts that a language model without SAE decomposition will store multiple concepts in the same neuron directions, leading to polysemanticity. SAEs are supposed to "undo" this superposition and recover monosemantic features.

You can measure monosemanticity quantitatively by examining the semantic diversity of the top-activating examples for a feature. If the examples cluster tightly around one concept in embedding space or in a human category taxonomy (for example, all examples are about cooking), the feature is likely monosemantic. If the examples split into multiple unrelated clusters (some about cooking, some about programming, some about sports), the feature is polysemantic.

Practically, you can use a language model to tag the top-$k$ examples with semantic categories, then compute the entropy of the category distribution:

where:

• $p_c$ is the fraction of top-$k$ examples belonging to semantic category $c$
• $H$ is the Shannon entropy (in nats if the natural logarithm is used, bits if log base 2 is used)
• Low $H$ indicates monosemanticity (examples cluster around a single concept)
• High $H$ indicates polysemanticity (examples are spread across many unrelated concepts)

A feature with all examples in one category has $H = 0$. A feature equally distributed across 10 categories has $H = \log 10 \approx 2.3$ bits. In practice, setting an entropy threshold of 1.5 bits (log base 2) over a 10-category taxonomy corresponds to one concept accounting for at least 70% of the top-$k$ examples, which is a reasonable operational definition of "near-monosemantic."

One subtlety is that the choice of category taxonomy affects the measured entropy. If your categories are very coarse (for example, "language about humans" vs. "language about non-humans"), many features will appear monosemantic. If your categories are very fine-grained (for example, distinguishing between "financial institutions" and "banking operations"), features that appeared monosemantic under a coarser taxonomy may turn out to be polysemantic. This sensitivity to the taxonomy is a methodological challenge, and it means that reported monosemanticity rates should be interpreted relative to the specific taxonomy used.

Beyond understanding individual features in isolation, a powerful complementary approach is to study the geometric structure of the feature space as a whole. Features are vectors in $\mathbb{R}^d$, and their relative positions, angles, and clusters carry information about the model's knowledge organization.

Two features $\mathbf{f}_i$ and $\mathbf{f}_j$ (taken as the decoder weight vectors) that represent semantically similar concepts tend to have higher cosine similarity:

For example, features representing different European capital cities will be closer to each other in feature space than to features representing programming language constructs. This geometric proximity is not accidental: it reflects the model's learned associative structure, shaped by the co-occurrence statistics in the training data.

This observation motivates clustering the feature space to discover "feature families": groups of semantically related features that occupy nearby regions of activation space. Clustering can be done with any standard algorithm (k-means, hierarchical clustering, UMAP-based visualization). The resulting clusters often correspond to recognizable semantic domains: a cluster of medical features, a cluster of mathematical notation features, a cluster of proper-name features for a particular language or culture.

Feature families also have a practical use in circuit analysis. If you are trying to understand a model's behavior on a specific topic (say, legal reasoning), you can start by finding the cluster of features in the "legal domain" family and then trace how those features interact with each other and with features in adjacent families.

Some of the most striking results in feature geometry concern linear subspaces. Just as in classic word embedding research we could find vector analogies like "king - man + woman = queen," SAE features sometimes exhibit analogous relational structure, but at a higher level of abstraction.

For instance, if you find a feature for "Paris is a capital city" and a feature for "France is the country containing Paris," there may be a consistent vector offset between features representing "X is a capital city" and "X is a major non-capital city" for various values of X. These structured relationships in feature space reflect the model's learned knowledge of systematic world-knowledge patterns: the same offset that relates "Paris" features to "Lyon" features also relates "Berlin" features to "Hamburg" features.

The existence of such relational structure has implications for how we think about what models have learned. It suggests that the model has organized facts into relational schemas represented as geometric structure in activation space, rather than memorizing individual facts in isolation. Discovering and characterizing these relational structures is one of the more intellectually exciting frontiers of feature geometry research.

As you train SAEs with different dictionary sizes on the same model, you observe a phenomenon called feature splitting and feature absorption. When you double the dictionary size, previously identified features often split into two or more finer-grained features that together cover the activation variance of the original feature. Conversely, when you reduce the dictionary size, related features merge into a single coarser feature.

This behavior provides a natural multi-resolution view of the model's representations. At low dictionary resolution, you see coarse semantic categories. At high resolution, you see fine-grained distinctions within those categories. The optimal dictionary size for a given interpretability question depends on how fine-grained your analysis needs to be: for understanding broad model behavior, a smaller dictionary may suffice; for understanding subtle within-category distinctions (for example, different registers of formal English), a larger dictionary is necessary.

Correlation between a feature and a concept is necessary but not sufficient for a valid interpretation. You also need to establish that the feature causally drives the behavior you have attributed to it. Two complementary causal methods are feature steering and feature ablation.

Feature steering involves artificially increasing a feature's activation and observing how the model's output changes. If a feature is tracking the concept "formal tone," then amplifying its activation should make the model's outputs more formal, regardless of what the input says. This kind of causal intervention is what distinguishes interpretability research from correlation analysis.

Operationally, you perform feature steering by computing the residual stream activation at a given layer in the normal forward pass, then adding a scaled multiple of the feature's decoder vector:

where:

• $\mathbf{x}$ is the original residual stream activation
• $\mathbf{d}_i$ is the decoder weight vector for feature $i$ (the direction in activation space the feature represents)
• $\alpha$ is the steering coefficient, which controls the strength of the intervention
• $\mathbf{x}_{\text{steered}}$ is the modified activation that will be passed to subsequent layers

The decoder vector is used for steering rather than the encoder vector because the decoder vector is the "concept direction" in residual stream space: it is the direction along which the model writes information associated with feature $i$. By adding $\alpha \mathbf{d}_i$ to the residual stream, you are injecting additional "evidence" for concept $i$ at that layer, as if the upstream circuit had activated the feature more strongly.

When steering succeeds in producing the expected behavior change, it provides strong evidence for the proposed interpretation. When it fails, it reveals either that the interpretation is wrong (the feature tracks a different concept) or that the feature is necessary but not sufficient for the behavior (other features must also be active for the behavior to manifest). Both outcomes are informative.

Steering experiments have produced some of the most vivid results in interpretability research. Anthropic's "Scaling Monosemanticity" paper demonstrated that steering on features associated with specific entities or concepts (such as a well-known person) could cause the model to produce text about that entity in a controlled way. More dramatically, steering on features associated with emotional states could shift the tone of model outputs along a spectrum from calm to agitated, giving direct evidence that these emotional features have a causal role in generation.

Feature ablation is the complementary operation to steering. Instead of amplifying a feature, you suppress it: set its activation to zero for a given input and measure the change in output. If a feature is necessary for a behavior, ablating it should cause the behavior to disappear or degrade.

Formally, ablation is applied by subtracting the feature's contribution from the residual stream:

where $a_i$ is the natural activation value that the feature achieves on the given input. This removes the feature's contribution exactly without affecting other features, as long as the SAE reconstruction is exact. In practice, SAEs have some reconstruction error, so ablation is approximate.

Ablation experiments reveal which features are necessary for a given behavior, while steering experiments reveal which features are sufficient. A feature that is both necessary (ablation kills the behavior) and sufficient (steering induces the behavior) has the strongest possible causal evidence for its proposed role. In practice, most features are either necessary or sufficient but not both, which is informative about the structure of the underlying circuit.

One important subtlety is that ablating a single feature rarely has a dramatic effect on a model's output. Most model behaviors are distributed across many features, so removing one is like removing a few bricks from a wall: the structure remains largely intact. This is consistent with good generalization: a model that relied on a single feature for a critical behavior would be fragile to noise in that feature. Meaningful ablation effects are typically found when you ablate entire groups of related features, or when you are studying a behavior that depends on a specific, rare feature with no redundant backup.

Let's build a practical toolkit for analyzing features from a trained SAE. We'll simulate the workflow using synthetic data, since running a real SAE requires substantial compute, but the analysis code translates directly to real SAE outputs.

We'll create synthetic activation data that mimics the output of a real SAE: a sparse matrix where most values are zero, with a few positive activations per token.

The activation matrix is highly sparse. Most entries are zero, and each token position activates only a small fraction of all features. This sparsity is the defining characteristic of SAE representations and is what makes them tractable for interpretation. If every feature fired on every token, you could not identify any meaningful pattern from the activation profile alone. The sparsity is both a computational convenience and an epistemic tool: it makes the question "why did this feature fire here?" answerable rather than trivially true everywhere.

Most features have low activation frequency, consistent with the Zipf-like distribution we built in. The top features fire much more often than the median, and a handful of features may be "dead" (never activating), which is a known failure mode in SAE training. In a real SAE trained on a large corpus, the frequency distribution would span roughly six orders of magnitude: from features that fire on nearly every token (structural or syntactic catch-alls) to features that fire on fewer than one in a million tokens (highly specialized, rare concepts).

In a real workflow you would look at actual text tokens; here we simulate by assigning each token a "concept label" drawn from a set of semantic categories. The key logic, however, is identical: sort by activation value, take the top $k$, and inspect the associated metadata.

The concept distribution reveals which semantic categories dominate the feature's top-activating examples. If one concept accounts for 70% or more of the top examples, the feature is likely monosemantic with respect to that concept. If the examples split roughly equally, the feature may be polysemantic or tracking a more abstract property that cuts across several surface-level categories. The ASCII bar chart representation is a quick visual heuristic: a feature with one very long bar is a good candidate for a clean label; a feature with many roughly equal bars needs further investigation.

The entropy score provides a quantitative summary of the concept-distribution analysis, allowing you to rank and filter features by their degree of monosemanticity without inspecting each one manually.

Features with entropy below 1.5 bits have a dominant concept covering at least 70% of their top examples; these are candidates for clean, monosemantic labels. Features above 2.5 bits distribute broadly across categories and require more careful investigation to determine whether they are polysemantic or whether the concept categories are too coarse for the feature's actual function. The mean dominant concept fraction gives another view of the same information: when this value is close to 1.0, most features are near-monosemantic; when it is close to $1/n_{\text{categories}}$, features are broadly polysemantic.

The side-by-side comparison illustrates the difference in activation profiles. The monosemantic feature (left) shows a narrow, peaked distribution: when it fires, it fires with a consistent strength, because it is responding to a single, well-defined concept. The polysemantic feature (right) shows a broader, flatter distribution: its activations are more variable because it is responding to multiple distinct contexts, each of which may activate it to a different degree.

Individual features do not operate in isolation. A model behavior is not produced by a single feature firing, but by a sequence of features activating across multiple layers, each influencing the next. Understanding a model behavior therefore requires understanding how features connect: which upstream features cause a given feature to activate, and which downstream features does it influence. The study of these connections is called feature circuit analysis.

A feature circuit is a subgraph of the transformer's computational graph in which the nodes are features (identified by SAE analysis) and the edges represent causal influences. An edge from feature $A$ to feature $B$ means that activating $A$ increases the probability that $B$ will activate, either directly (through the residual stream) or indirectly (through intermediate computations in attention or MLP layers).

The concept was formalized in the paper "Sparse Feature Circuits" by Marks et al. (2024), which showed that many model behaviors can be explained by small, compact circuits involving tens of features out of millions. For instance, the behavior "refuse to complete a toxic prompt" might be explained by a circuit in which a toxicity-detection feature in early layers activates a refusal-decision feature in a middle layer, which in turn modulates the output distribution in later layers to suppress harmful completions.

The appeal of the circuit framing is that it makes model behavior legible. Instead of asking "why did the model generate this output?", you can decompose the question into smaller, more tractable sub-questions: which features were active, which upstream features caused them, and which output logits did they influence? If the circuit is small, the answer is tractable. If the circuit is large, you have learned that the behavior is distributed and cannot be explained by a simple causal story, which is also informative.

This is why feature circuits are considered one of the most promising directions in mechanistic interpretability. They promise to reduce the analysis of arbitrarily complex model behaviors to a manageable graph problem, connecting the vocabulary of interpretable features to the downstream computation that produces model outputs.

The standard empirical method for identifying which features matter for a given behavior is activation patching, which we explored in an earlier chapter on interpretability foundations. In the context of feature circuits, activation patching works as follows. You define a "clean" input that produces the target behavior and a "corrupted" input that does not. You run both inputs through the model, storing activations at every SAE feature. Then you patch individual feature activations one at a time, replacing the clean value with the corrupted value, and measure how much the output changes.

Features for which patching causes a large change in output are causally relevant to the behavior. By composing patches at different layers, you can trace the causal chain from the input to the output, identifying the full circuit. The advantage of this approach over pure gradient attribution is that it is causal rather than correlational. A feature that is merely co-activated with the behavior (but not causally relevant) will produce little change when patched. A feature that is part of the circuit will produce a large change. This distinction matters enormously in practice: many features are activated by the same input distribution that triggers a behavior without being part of the mechanism that produces it.

Activation patching at the feature level is more computationally demanding than patching at the residual stream level, because you must run the SAE encoder and decoder for every position and every layer. In a large model with dozens of layers and millions of SAE features, this can require substantial compute. But the resulting circuits are far more interpretable because the nodes have meaningful labels, rather than being abstract residual stream vectors.

A practical refinement is hierarchical patching: rather than patching individual features, you first identify important layers (or attention heads) using residual-stream-level patching, and then apply feature-level patching only within those layers. This two-pass approach dramatically reduces the computational cost while preserving the interpretability of the final circuit.

For a feature at layer $l$, there are two symmetric questions about its causal role:

1. Upstream: Which features in earlier layers have the highest causal influence on this feature's activation? Answering this question traces the feature's causes.
2. Downstream: Which features in later layers does this feature causally influence? Answering this question traces the feature's effects.

These questions are addressed using gradient-based attribution. The upstream influence of feature $j$ at layer $l-1$ on feature $i$ at layer $l$ can be measured as:

where:

• $a_i^{(l)}$ is the activation of feature $i$ at layer $l$
• $a_j^{(l-1)}$ is the activation of feature $j$ at layer $l-1$
• $\frac{\partial a_i^{(l)}}{\partial a_j^{(l-1)}}$ is the gradient of feature $i$'s activation with respect to feature $j$'s activation, measuring how sensitive $i$ is to $j$
• The product of gradient and activation captures both the sensitivity and the actual magnitude of the source feature's contribution

This is a first-order linear attribution and is closely related to the "gradient times input" attribution method used in other interpretability contexts. For nonlinear interactions (for example, when an attention gate controls whether a feature's influence is transmitted), higher-order terms would be needed, but the first-order version already identifies the most important direct upstream contributors for most features.

To trace the full upstream circuit of feature $i$, you apply this attribution recursively: compute the upstream attributions for feature $i$, then compute the upstream attributions for the most important upstream features, and so on. You stop when you have traced back to the embedding layer or when the attribution scores become negligibly small. The resulting tree of attributions is the feature's upstream circuit.

The transformer's residual stream provides the natural framework for tracing feature circuits. Recall from our earlier discussion of transformer architecture that the residual stream at position $t$ and layer $l$ accumulates contributions from all previous attention heads and MLP layers:

where:

• $\mathbf{x}^{(0)}$ is the initial token embedding
• $\mathbf{a}^{(l')}$ is the summed output of all attention heads at layer $l'$
• $\mathbf{m}^{(l')}$ is the output of the MLP at layer $l'$

Each SAE applied to a particular layer $l$ operates on $\mathbf{x}^{(l)}$ and decomposes it into a sparse linear combination of feature vectors. When tracing circuits, we track how specific features in the decomposition at layer $l$ were caused by specific components (particular attention heads or MLP neurons) at earlier layers.

The key tool for this is the "direct path" decomposition: for a given feature $i$ at layer $l$, we ask which components of $\mathbf{x}^{(l)}$ most contributed to its activation. Because the residual stream is additive, we can decompose $\mathbf{x}^{(l)}$ into its constituent contributions from each upstream layer and component:

Each term in this sum is the attribution of a specific upstream component to feature $i$'s pre-activation. Components with large positive terms are the main contributors to the feature's activation; components with large negative terms suppress it.

Attention heads are a particularly important node type in feature circuits. An attention head reads from the residual stream at query and key positions, computes attention weights over the sequence, and writes a value vector back to the residual stream at each position. If we have SAE features at both the input and output of an attention layer, we can measure how strongly each head's output contributes to each output feature.

Let $\mathbf{v}_h$ be the value vector written by attention head $h$ at token position $t$. The contribution of head $h$ to output feature $i$ is:

where $\mathbf{W}_{\text{enc}}^{(i)}$ is the encoder weight vector for feature $i$. This is a direct measure of whether head $h$'s output aligns with the direction of feature $i$ in residual stream space. A large positive value means head $h$ is writing in the direction that activates feature $i$; a large negative value means it is suppressing the feature.

By computing this for all heads and all features, you get an attribution matrix. Heads that strongly write in the direction of a feature are part of that feature's upstream circuit. This connects the SAE feature analysis directly to the attention pattern analysis: you can ask which attention head is important and which source positions that head is attending to when it writes the feature-activating information. The full causal chain then runs from a source token, through attention weights that select that token, through a value computation that extracts information from the source token's representation, to the residual stream update, to the SAE feature activation, to the downstream features and ultimately to the model's output logits.

Let's implement a simplified version of feature circuit analysis. We'll work with a toy model that mimics the residual stream structure, using the same synthetic SAE activations we created earlier.

For each active feature at the final layer, we compute how much each attention head at each layer contributed to it. The attribution is computed as the mean absolute projection of each head's output onto each feature's encoder direction, averaged across all token positions.

The attribution analysis reveals that not all heads contribute equally to the target feature. The top-contributing (layer, head) pairs represent the upstream circuit for this feature: these are the computation steps that write information into the residual stream in the direction of the feature's encoder vector. In a real model, this would tell you that, for example, "Layer 3, Head 2 is responsible for writing the capital-city relationship feature, and it does so by attending from the country-name token to the capital-city token."

Features that frequently activate together on the same tokens are likely functionally related. Co-activation can arise for several reasons: two features may be part of the same circuit and tend to be activated together by the same upstream cause, or they may be tracking different aspects of the same input context, or they may be redundant features covering similar concepts (a common occurrence when the SAE dictionary is larger than the effective dimensionality of the feature space).

We measure co-activation with pointwise mutual information (PMI), which controls for the marginal frequencies of each feature. Two frequent features will have many co-activations by chance alone; PMI normalizes this away by comparing the observed co-activation rate against the rate expected under independence.

The PMI heatmap is a useful first filter for identifying which features are worth investigating together in a circuit analysis. Pairs with high PMI that are also semantically similar (based on their feature labels) are strong candidates for being part of the same functional circuit. Pairs with high PMI but dissimilar labels may indicate an interpretation error in one of the features, or a compound concept that spans both labels.

Feature interpretation is a young and rapidly evolving field. The methods described in this chapter are already producing remarkable results, but several significant challenges limit the conclusions that can currently be drawn from SAE analyses.

Automated interpretability pipelines produce natural language descriptions, but natural language is imprecise and context-dependent in ways that can mislead. A feature described as "mentions emotions" may instead be tracking a subtle distributional property that the description inadequately captures: perhaps it fires specifically on first-person expressions of negative affect in formal writing, a pattern that "mentions emotions" fails to represent. The autointerpretability score measures correlation between a description-based simulator and actual activations, but correlation is not causation or completeness.

A description could score well on the simulation task but still be fundamentally wrong if the proxy task is too easy, or if the description happens to be coextensive with the true pattern even though the concepts are unrelated. Conversely, a correct but complex description might score poorly because the simulator model cannot reliably implement the described rule in its one-shot simulation task.

The deeper problem is that there is no ground truth for feature interpretation. Unlike in supervised learning, where the correct answer is defined by a label, the "correct" interpretation of a feature direction in activation space is not formally defined. Features encode statistical patterns in the training data, and those patterns do not always have clean, human-interpretable labels. The best we can say is that a description is more or less useful, more or less predictive, and more or less consistent with the available evidence.

Even if every individual feature is well-interpreted, understanding model behavior still requires understanding which features are active for a given input, how they interact across layers, and what the model ultimately concludes. Feature circuits partially address this, but tracing a complete circuit through all layers of a modern large language model with dozens of layers and billions of parameters is computationally demanding. The resulting circuits can be complex enough to resist easy interpretation even after all the nodes and edges have been identified.

There is also a compositionality question: many model behaviors emerge from the interaction of hundreds or thousands of features, not from a small, clean circuit. The transformer's attention mechanism can create complex, input-dependent interactions between distant token positions. When the functional circuit for a behavior involves hundreds of features interacting across twenty layers, circuit analysis may reveal all the components without providing a satisfying explanation of how they work together. The circuit is correct but not comprehensible.

This problem is sometimes called the "legibility gap": the description of a circuit may be technically accurate (it correctly identifies all the relevant features and edges) while still being too complex for a human to understand. Bridging the legibility gap may require finding higher-level abstractions that summarize circuit structure, analogous to how a software engineer understands a program in terms of functions and modules rather than individual machine instructions.

A well-trained SAE should recover monosemantic features, but in practice SAE training involves tradeoffs that compromise this ideal. The sparsity penalty ($L_1$ regularization on activations) pushes features toward monosemanticity by encouraging each feature to fire only when a specific concept is present. But the sparsity penalty also introduces a systematic bias: it causes the SAE to underestimate activation values, because the optimizer knows it will be penalized for any positive activation and therefore sets the threshold conservatively.

When many features are active simultaneously (as happens for complex, information-rich tokens), the SAE must choose how to distribute the representation across available features. This distribution is shaped by the features that best match the input and by the interaction between the sparsity penalty and the reconstruction objective. The result is that some features are shaped by their primary concept and by their secondary role in reducing reconstruction error for other inputs. The feature directions are not purely monosemantic but are subtly distorted by the learning process.

Detecting and correcting for this distortion is an active research problem. One diagnostic approach is to train multiple SAEs with different dictionary sizes and sparsity levels on the same model, and check whether the identified feature directions are stable across training runs. Stable directions are more likely to represent stable structure in the model's representations; unstable directions may be artifacts of a particular SAE training configuration.

The computational cost of interpreting every feature scales with both model size and dataset size. Collecting top-$k$ activating examples for millions of features requires running forward passes over a corpus large enough to see meaningful statistics for rare features. For a feature that fires on one in a million tokens, you need at least tens of millions of tokens just to gather a reliable top-$k$ set, and potentially billions of tokens to see the full distribution of contexts where the feature fires.

This creates a fundamental resource inequality. Feature interpretation at the quality described in research papers from Anthropic and other major labs requires infrastructure at roughly the scale of the original model training: the same hardware, similar compute costs, and large engineering teams. Smaller research groups and individual practitioners cannot conduct this kind of exhaustive analysis for the models they work with.

There is an ongoing effort to democratize feature interpretation. Projects like Neuronpedia provide public dashboards for exploring SAE features in open-source models, allowing practitioners to browse top-activating examples and read automated descriptions without running large-scale inference themselves. The EleutherAI and other open research groups have released SAE weights trained on open-source models like Pythia and LLaMA, enabling the community to conduct feature analysis without training from scratch. These resources lower the barrier to entry significantly, though they do not resolve the fundamental scaling problem for frontier models.

A recurring challenge in feature interpretation is the absence of a reliable evaluation framework. It is relatively straightforward to produce a feature label and much harder to verify that the label is correct and complete. Current evaluation methods include:

• Held-out simulation: testing whether the description predicts activations on new examples not used to generate the description. This measures precision but not recall: a description that fires on exactly the right tokens but also misses many tokens where the feature fires will score well on precision but poorly on recall.
• Ablation studies: removing or suppressing the feature's activation and checking whether the model's behavior changes as predicted. This tests causal relevance but requires knowing which behavior to monitor, which is itself a non-trivial interpretability question.
• Steering experiments: amplifying the feature and checking whether the model's outputs shift in the expected direction. This provides a complementary causal test to ablation, but steering may produce side effects if the feature direction is not perfectly isolated from other features.
• Human agreement: having multiple researchers independently label the same features and measuring inter-annotator agreement. This tests labeling consistency but not accuracy: two researchers can agree on an incorrect label.
• Cross-model comparison: checking whether the same feature direction (or a close neighbor) appears in independently trained models. Convergent features across training runs are stronger candidates for representing semantic content rather than artifacts of a particular training seed.

None of these methods is fully satisfactory in isolation. A feature that passes all five tests is a strong candidate for a correct interpretation, but even then the interpretation may be incomplete (covering the primary concept but missing secondary roles), imprecise (using a coarser category than the true concept), or wrong in rare edge cases. Building a rigorous evaluation standard for feature interpretation is an active research area, closely tied to the broader question of what "understanding a model" should mean.

When running your own feature interpretation pipeline, the key choices are:

• k (top examples): The number of top-activating examples to inspect per feature. Using 20 to 50 examples provides enough evidence to identify a dominant theme without overwhelming the annotator or the automated interpreter.
• Activation threshold: The minimum activation value to count a token as active. Setting this too low includes noise from near-zero activations; setting it too high misses weak but valid activations for rare concepts.
• Entropy threshold for monosemanticity: Values below 1.5 bits (log base 2) over a 10-category space correspond to one concept dominating 70% or more of the examples. Adjust this threshold based on the granularity of your category taxonomy.
• Attribution depth: How many layers back to trace when building feature circuits. Two to three layers captures the most direct causal structure; deeper tracing reveals more distant context but is computationally expensive and produces noisier attributions.
• PMI threshold for co-activation graphs: Filter edges below a normalized PMI of 0.5 to focus on the strongest co-activation relationships and avoid spurious correlations from common syntactic contexts.
• Steering coefficient: The scale $\alpha$ for feature steering experiments. Start small (0.1 to 1.0 times the typical active activation value) and increase until an effect is visible. Very large steering coefficients can move the residual stream far outside the training distribution and produce incoherent outputs.

Feature interpretation turns the abstract output of a sparse autoencoder into human-readable understanding. The key steps are:

• Activation pattern analysis: identify which inputs activate each feature, compute frequency statistics, and study activation histograms to distinguish monosemantic features (narrow, peaked distributions, low entropy) from polysemantic ones (broad distributions, high entropy).
• Feature naming: assign human-readable labels through careful manual inspection for small dictionaries, or through automated interpretation pipelines for large SAEs. Calibrate descriptions using held-out simulation to ensure they are predictive rather than merely descriptive.
• Monosemanticity measurement: use the Shannon entropy of the top-activating examples' concept distribution to quantify how cleanly a feature maps to a single concept, using thresholds of 1.5 bits (near-monosemantic) and 2.5 bits (polysemantic) as practical guides.
• Feature geometry: study how feature vectors cluster in activation space, identifying feature families (semantically related groups), linear relational structure (analogies between feature directions), and the effects of dictionary size on feature splitting and absorption.
• Causal validation: go beyond correlation by using feature steering (amplifying a feature and checking that the expected behavior appears) and feature ablation (suppressing a feature and checking that the behavior degrades). Together these test whether a feature is sufficient and necessary for a behavior.
• Feature circuits: trace causal connections between features across layers using attribution methods, revealing the functional pathways through which information flows from input tokens to output logits.

The limitations of current feature interpretation methods are substantial: descriptions are hard to validate rigorously, complete circuit analysis is computationally expensive, and the SAE training process itself introduces biases that compromise monosemanticity. Despite these challenges, feature interpretation has already produced specific findings into how language models encode linguistic structure, world knowledge, and reasoning patterns.

The next chapter on Mechanistic Interpretability builds on these foundations in a more targeted way. Where feature interpretation asks "what does each feature represent?", mechanistic interpretability asks "how do groups of features implement specific algorithms?" We will see how circuits involving attention heads and MLP layers can be reverse-engineered to reveal that transformers implement surprisingly structured computations, from simple lookup tables to multi-step reasoning chains, using the same building blocks we have studied throughout this handbook.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about feature interpretation in sparse autoencoders.