Feature Interpretation › Feature Activation Patterns

The most direct way to understand a feature is to look at which inputs activate it. Before you can name a feature or understand its role in a circuit, you need to observe the raw activation signal: when does it fire, how strongly, and over what range of contexts?

A feature direction $\mathbf{f} \in \mathbb{R}^d$ fires on a token $t$ in context $C$ with activation:

$$a = \text{ReLU}(\mathbf{W}_{\text{enc}}^{(i)} \cdot \mathbf{x} - b_i)$$

where:

• $\mathbf{W}_{\text{enc}}^{(i)}$ is the $i$-th row of the SAE encoder weight matrix, representing the feature direction
• $\mathbf{x}$ is the residual stream activation at the position of token $t$
• $b_i$ is the learned bias (the pre-activation threshold) for feature $i$
• $a$ is the activation value, zero when the feature does not fire and positive when it does

The ReLU function serves two purposes here. First, it enforces non-negativity: activations represent the degree to which the feature is present, and a negative projection does not represent "negative presence" in a useful sense. Second, the bias term $b_i$ learned during SAE training acts as a threshold below which the feature is considered absent. The SAE training process learns this threshold jointly with the encoder and decoder weights, so the threshold is calibrated to keep the representation sparse while minimizing reconstruction error.

To study activation patterns empirically, you collect the top-$k$ activating examples from a large corpus for each feature. The standard procedure is to run the model over a large dataset, record the activation value $a$ for each token position, and then sort by $a$ to find the examples where the feature fires most strongly. For practical feature analysis, this typically requires running inference over millions to billions of tokens, which is computationally demanding but feasible on modern hardware for models up to a few billion parameters.

The "max-activating examples" approach, popularized by Anthropic and used extensively in the paper "Towards Monosemanticity," shows the most and least activating contexts for each feature. The idea is simple: if a feature is tracking a coherent concept, its top-$k$ activating tokens and surrounding context should cluster around a recognizable theme.

For example, a feature that fires highly on tokens like "Paris", "London", "Berlin" when used as subjects of verbs like "is the capital of" is revealing information about capital-city relationships. A feature that fires on words like "however", "nevertheless", "despite" is tracking concessive discourse connectives. A feature that fires on tokens in Python docstrings but not in prose is tracking a structural property of source code. These patterns emerge directly from looking at examples, without any supervision or pre-specified concept taxonomy.

The key insight is that you can observe the distribution of activating examples and use it to formulate a hypothesis about the feature's function. This hypothesis can then be tested by finding examples that should activate it (if it tracks concept X, does it fire on all instances of X?) and examples that should not (is there cross-activation on unrelated concepts?). The process is fundamentally empirical and iterative: your first hypothesis will often be wrong or incomplete, and revising it requires going back to more examples.

In practice, researchers display max-activating examples with the activating token highlighted and several tokens of surrounding context shown. The activation value is color-coded by intensity: tokens that cause stronger activations are shown with deeper color. This visualization immediately reveals whether the feature is tracking the highlighted token itself, a property of its grammatical context, or something in the broader paragraph context. A feature that fires when "bank" is the highlighted token but only in certain sentence contexts is doing something more sophisticated than simply recognizing the word form.

Features in a well-trained SAE have a characteristic activation histogram: a large mass of zeros (the sparse regime, where the feature does not fire) and a long tail of positive activations (the active regime). The shape of this histogram is diagnostic. A feature that fires broadly and weakly may be polysemantic or a catch-all, serving as a residual absorber for variance that the SAE cannot explain with sharper features. A feature that fires rarely but intensely is likely tracking a specific, rare concept, such as a technical term, a named entity, or a particular grammatical construction that appears infrequently in the training corpus.

You can quantify this with a few statistics:

• Activation frequency: the fraction of tokens in the corpus for which $a > 0$. Well-interpreted features often have frequencies ranging from $10^{-4}$ to $10^{-1}$.
• Mean active activation: the average value of $a$ conditioned on $a > 0$. A high mean active activation relative to the threshold indicates a feature that fires strongly when it fires at all.
• Log activation rank: the rank of a feature by frequency. Frequency distributions across a trained SAE are approximately log-uniform, similar to Zipf's law in natural language. This is not a coincidence: the training corpus itself follows approximately Zipfian statistics, so features that track concept frequency inherit this distribution.

Features near the high-frequency end of the spectrum deserve special attention. Very frequent features often turn out to track basic syntactic or structural properties: "token is part of a number", "token appears at the end of a sentence", "token is in all-caps". These are real features of language, but they are not particularly informative about the model's higher-level knowledge. Very rare features may track niche concepts, or they may be "dead features" that never fire due to poor initialization during SAE training.

Dead features deserve special mention. In a large SAE dictionary, a substantial fraction of features may have activation frequency effectively zero over any reasonable-sized corpus. This happens because SAE training with an $L_1$ sparsity penalty can collapse certain feature directions: the optimizer finds it easier to set a feature's activation to zero for all inputs than to learn a specialized direction that would fire infrequently. Detecting dead features is important for quality control, as they represent wasted capacity in the dictionary. Heuristically, features with activation frequency below $10^{-6}$ over a corpus of one million tokens should be treated as dead.

An important distinction is the difference between a feature's token-level activation and its context-level meaning. SAEs typically operate on residual stream vectors at individual token positions, so the activation $a$ is computed separately for each token. But the meaning of an activation often depends heavily on the surrounding context.

For instance, a feature might fire strongly on the token "bank" only in the context of financial sentences ("I deposited money at the bank"), not when "bank" refers to a river bank or a data bank. This context-sensitivity is a sign that the model has already resolved polysemy before the SAE sees the activation: the residual stream carries the contextual interpretation of the token, not just its raw identity. The attention heads in earlier layers have already integrated the surrounding tokens to produce a context-sensitive representation, and the SAE operates on that already-contextualized vector.

This distinction is important for interpretation in two ways. First, when studying max-activating examples, you should always display enough surrounding context to understand what the model was computing, in addition to the highlighted token. A feature described as "the token 'bank'" is almost certainly wrong: it is more accurately described as "financial institutions in context." Second, it means that features at later layers tend to be more abstract and context-sensitive than features at earlier layers, which are closer to the raw token embeddings and thus more often track surface-level properties like token identity, capitalization, or word length.

The contrast between early-layer and late-layer features is one of the more striking empirical findings in SAE research. Early-layer features tend to look like enhanced versions of word embeddings: they capture lexical meaning, part of speech, and basic morphological properties. Late-layer features often capture more abstract relational properties: whether this token is part of an argument structure, whether the current text is describing a causal relationship, whether the model is in the middle of a list or a narrative.