Feature Interpretation › Feature Naming

Identifying what a feature represents requires assigning it a human-readable label. Feature naming is the bridge between raw activation patterns and interpretable descriptions, and it is both more art than science and more science than art: there are principled methods for evaluating descriptions, but the initial hypothesis generation is largely a creative act.

In the original work on sparse autoencoders applied to small transformer models (including Anthropic's work on toy models of superposition and the subsequent "Towards Monosemanticity" paper), researchers assigned names to features manually by inspecting the top-activating examples. This process is labor-intensive but highly reliable when done carefully.

The workflow for manual interpretation involves several steps. First, for each feature, you retrieve the top 20 to 50 examples (or tokens) by activation strength. You read these examples carefully, giving special attention to both the token highlighted with maximum activation and the surrounding context. Second, you identify a common theme across the examples. This requires relevant linguistic and domain knowledge: a researcher without knowledge of Python syntax will struggle to label features that fire on docstrings or type annotations. Third, you propose a short label that is precise and falsifiable, such as "Hebrew text", "Python variable names", or "negative sentiment adjectives." Finally, you validate the label by checking for false positives (do any top-activating examples not match the description?) and false negatives (do known instances of the concept fail to activate the feature?).

Manual naming is valuable because experienced researchers notice nuances that automated systems miss. They might notice that a feature fires on both formal titles ("Dr.", "Prof.") and on abbreviations in scientific notation ("mL", "cm"), which would lead to a more careful decomposition rather than a single imprecise label. They might notice that a feature fires more strongly on the first occurrence of a named entity in a passage than on subsequent mentions, revealing something about the model's coreference tracking. These observations require the interpretability researcher to bring substantial world knowledge and linguistic intuition to the task, which cannot be fully automated.

The main drawback is scale. Modern SAEs trained on large language models have millions of features. Inspecting them individually at the level of quality described above is not feasible. Researchers at major labs have noted that careful manual labeling of a single feature can take several minutes, which means that a million-feature SAE would require millions of person-hours to label manually. This scaling gap is the primary motivation for automated interpretation pipelines.

To scale feature naming, researchers use language models themselves as automated interpreters. The general pipeline converts the problem of feature labeling into a language modeling task: given the top-$k$ activating examples for a feature, ask a language model to describe what they have in common.

The pipeline has three stages:

1. Collection: For each feature, retrieve the top-$k$ activating examples from the corpus, preserving both the highlighted token and its context window. The number of examples $k$ is typically between 10 and 30: too few and the description generator has insufficient evidence; too many and the prompt becomes unwieldy and expensive.

2. Generation: Format the examples as a prompt that asks a language model to describe the common pattern. A typical prompt might present the examples with activation values and ask: "Based on these contexts, what pattern does this feature appear to track? Provide a concise description in one or two sentences."

3. Calibration: Score the proposed description by testing it as a predictor on held-out examples. The scoring step is critical and is what distinguishes a rigorous automated pipeline from simple summarization.

The calibration step deserves elaboration. A description that says "mentions emotions" is nearly vacuous: it matches a broad swath of text and does not make useful predictions about which specific contexts will cause the feature to fire. The ideal description makes precise, falsifiable predictions. To measure this, the calibration stage uses a simulator: another language model is given the proposed description and asked, for each token in a new passage, whether the feature should be active at that position. The simulated activations are then compared against the actual feature activations, and the correlation is used as the calibration score.

Anthropic's automated interpretability pipeline, used in their feature analysis of Claude models and in the open-source Neuronpedia tool, formalizes this into three steps: simulation, description, and calibration. The simulator takes a proposed description and predicts which tokens in a new context should cause the feature to fire; the calibration metric measures the Pearson correlation between the simulator's predicted activations and the actual activations from the SAE.

The automated interpretation pipeline reveals a striking finding: a large fraction of SAE features trained on frontier language models have interpretable, human-understandable descriptions with autoint scores well above chance. In Anthropic's analysis of a one-billion-parameter model, a majority of features were judged as cleanly monosemantic, with a smaller fraction covering polysemantic or unclear behaviors. This result supports the core theoretical prediction of the superposition hypothesis: that SAE decomposition should recover features that are more semantically coherent than raw neurons.

An important practical observation is that the "vocabulary" of features changes qualitatively across layers of the transformer. When you apply SAEs to multiple layers of the same model and analyze the resulting features, you see a rough semantic hierarchy emerge.

In the first few layers, features tend to be near-syntactic: they track properties like part of speech, capitalization, whether a token is numeric, whether it appears in a URL, whether it is a punctuation mark. These features are relatively easy to interpret and tend to have high autoint scores, because the mapping from token identity to concept is fairly direct.

In the middle layers, features shift toward local semantic properties: named entity types (person, organization, location), sentiment polarity, domain indicators (medical, legal, scientific), discourse structure markers. These features are somewhat harder to interpret because they depend on context rather than the token alone, but they are still often cleanly monosemantic.

In the final layers, features tend to be abstract and relational: they may capture properties like "this is the conclusion of an argument", "this is a counterfactual statement", "this token is part of a reasoning chain." These features are the hardest to interpret and often have lower autoint scores, showing the difficulty of expressing abstract relational properties in natural language.

This layer-wise progression is an empirical observation and an important guide for practitioners. If you are trying to understand a model's behavior on a specific task, the layer at which you apply the SAE matters enormously. Features at early layers will tell you about input preprocessing and token-level properties. Features at late layers will tell you about the model's high-level reasoning and output planning. For most interpretability questions about model behavior, the most informative features tend to be in the middle to late layers.

A core goal of feature interpretation is to measure the degree of monosemanticity: how well a single feature corresponds to a single concept. The theory of superposition, discussed in the previous chapter on SAEs, predicts that a language model without SAE decomposition will store multiple concepts in the same neuron directions, leading to polysemanticity. SAEs are supposed to "undo" this superposition and recover monosemantic features.

You can measure monosemanticity quantitatively by examining the semantic diversity of the top-activating examples for a feature. If the examples cluster tightly around one concept in embedding space or in a human category taxonomy (for example, all examples are about cooking), the feature is likely monosemantic. If the examples split into multiple unrelated clusters (some about cooking, some about programming, some about sports), the feature is polysemantic.

Practically, you can use a language model to tag the top-$k$ examples with semantic categories, then compute the entropy of the category distribution:

$$H = -\sum_{c} p_c \log p_c$$

where:

• $p_c$ is the fraction of top-$k$ examples belonging to semantic category $c$
• $H$ is the Shannon entropy (in nats if the natural logarithm is used, bits if log base 2 is used)
• Low $H$ indicates monosemanticity (examples cluster around a single concept)
• High $H$ indicates polysemanticity (examples are spread across many unrelated concepts)

A feature with all examples in one category has $H = 0$. A feature equally distributed across 10 categories has $H = \log 10 \approx 2.3$ bits. In practice, setting an entropy threshold of 1.5 bits (log base 2) over a 10-category taxonomy corresponds to one concept accounting for at least 70% of the top-$k$ examples, which is a reasonable operational definition of "near-monosemantic."

One subtlety is that the choice of category taxonomy affects the measured entropy. If your categories are very coarse (for example, "language about humans" vs. "language about non-humans"), many features will appear monosemantic. If your categories are very fine-grained (for example, distinguishing between "financial institutions" and "banking operations"), features that appeared monosemantic under a coarser taxonomy may turn out to be polysemantic. This sensitivity to the taxonomy is a methodological challenge, and it means that reported monosemanticity rates should be interpreted relative to the specific taxonomy used.