Sparse Autoencoders › Applying SAEs to Language Models

The most common application of SAEs to language models is to train them on the residual stream activations at a specific layer. Given a trained language model, you collect a large dataset of text, run it through the model to record activation vectors at layer $\ell$, and then train an SAE on those vectors. The SAE is trained completely separately from the language model, which remains frozen throughout. This is important: the language model's weights are not updated, so the SAE learns to decompose existing representations rather than shaping how the model represents information.

This decoupled training strategy has both advantages and limitations. The advantage is that you can train many different SAEs (varying the hookpoint, dictionary size, or $\lambda$) without retraining the expensive base model. The limitation is that the SAE can only represent what is already in the activation space; it cannot discover features that the model has not already learned to encode.

SAEs can be trained on different components of a transformer. The choice matters because different components carry different kinds of information:

• Residual stream: The most common choice. The residual stream at each layer is the "main highway" of information, and SAEs trained there tend to learn the cleanest features. Because the residual stream aggregates contributions from all attention heads and MLP layers up to that point, features at later residual stream positions reflect the cumulative computations of all preceding layers.
• MLP output: The MLP sublayer is where most knowledge storage is thought to occur. SAEs trained on MLP outputs often learn factual and entity-level features, including features that correspond to specific persons, places, and concepts from the training data.
• Attention output: SAEs trained on attention output layers learn features related to syntactic relationships and positional information, such as subject-verb agreement patterns and co-reference.
• Attention query/key: Less common, these can reveal patterns in how attention heads select information. Analysis here may help understand which features cause an attention head to attend to a particular token.

The choice of hookpoint affects both what features are learned and how they can be used for downstream analysis. Residual stream SAEs are the most general: because the residual stream aggregates contributions from all earlier layers, features there can reflect both syntactic and semantic information simultaneously. MLP SAEs tend to be more focused on factual knowledge and less on syntactic structure, which can be an advantage when you want to study how the model stores and retrieves facts.

Different layers also yield different types of features. Earlier layers tend to encode lower-level syntactic properties (part of speech, morphology, local context), while later layers encode higher-level semantic properties (entity type, argument structure, discourse relations). This layer-by-layer progression mirrors what has been found using probing classifiers. It also suggests a natural experimental strategy: if you want to understand when a particular concept is computed, you can train SAEs at many layers and observe at which layer a feature for that concept first appears with high fidelity.

Training high-quality SAEs requires enormous amounts of data. The SAE is trained on a distribution of activation vectors, and every forward pass through the language model generates one activation per layer per token. A typical training run for a large-scale SAE uses billions of tokens. For a model with $d = 4096$ and a dictionary size of $m = 16384$, this means processing on the order of $10^{12}$ floating-point numbers across the training run.

Fitting the SAE weights alone does not explain the scale requirement. The SAE has only $md + d + m$ parameters (encoder weights, pre-bias, and encoder bias), which for $d = 4096$ and $m = 16384$ amounts to roughly 67 million parameters. You do not need billions of examples to fit 67 million parameters in a traditional sense. The need for scale comes from feature coverage: rare features, such as a feature for a specific historical event or a niche programming pattern, might activate on only a tiny fraction of tokens. If the training dataset does not contain enough examples of that pattern, the SAE may fail to learn a coherent feature for it. The SAE needs to see each feature activate enough times (perhaps hundreds of times at minimum) for the gradient signal to shape the corresponding dictionary vector reliably.

For practical intuition: if feature $j$ fires on $0.001\%$ of tokens (one per 100,000), you need at least 100 million tokens to see that feature fire 1,000 times. For features with activation rates of $0.0001\%$ or lower, you need 10 billion or more tokens. Real language models have millions of features at varying activation rates, and ensuring adequate coverage of even moderately rare features is the primary driver of the data scale requirement.

Anthropic's publicly released SAEs for Claude's Sonnet model, for example, were trained with hundreds of billions of activation samples and contain millions of features. Google DeepMind's Gemma Scope project released SAEs for the Gemma 2 model family with similar scale. These large-scale training efforts are expensive but produce features with high fidelity and semantic coherence.

Once an SAE is trained, how do you know if a feature is meaningful? The standard approach is max-activating example analysis: for each feature $j$, collect the top-k tokens (or token contexts) that produce the highest activation $z_j$. If those contexts share a coherent semantic theme (e.g., the top-activating contexts for a feature are all city names), the feature is considered interpretable.

This approach is analogous to the receptive field analysis used in neuroscience: to understand what a neuron encodes, you find the stimuli that drive it most strongly. A neuron in visual cortex that responds maximally to vertical edges oriented at 45 degrees is interpretable; a neuron that responds to a mix of unrelated visual patterns is not.

This approach is inherently subjective, so researchers have also developed automated interpretability methods. These typically use a language model to read the top-activating contexts for a feature and generate a natural language description (e.g., "This feature activates for mathematical formulas in LaTeX"), then verify that description by checking whether the feature activates for new contexts that match the description and does not activate for contexts that do not. The verification step is what distinguishes a good feature label from a spurious coincidence in the top-activating examples.

Beyond max-activating examples, researchers also examine the feature's effect on model outputs by activation steering: artificially increasing or decreasing $z_j$ and observing how the model's predictions change. If boosting a "feature for Paris" feature increases the probability of tokens like "France", "Eiffel Tower", and "Seine", that is strong evidence the feature encodes a concept related to Paris. If boosting a "feature for Paris" has no coherent effect on outputs, the feature label may be wrong or the feature may not be causally relevant to the model's computations.

Large-scale SAE experiments on frontier language models have produced a remarkable catalog of learned features. The sheer variety of what these features represent is itself an important empirical finding.

Some features are highly specific to surface-level token patterns. There are features that activate for text in specific scripts (Arabic, Cyrillic, Korean), features that fire for specific programming language keywords, and features that respond to punctuation in specific positions. These are relatively unsurprising: surface-level regularities are easy to learn and are presumably useful to many downstream computations.

More striking are features that appear to encode abstract semantic concepts. Anthropic's analysis of Claude models identified features for specific named entities, where a single feature would activate for all mentions of a particular person across different contexts, names, and phrasings. A feature for "Albert Einstein" might activate for "Einstein", "the physicist who developed relativity", "E = mc^2", and "the author of the special theory of relativity". The feature is not tracking a surface pattern (these are very different strings) but an abstract identity.

There are also features that track relational and structural properties of text. Features for "beginning of a list", "the token following a colon", "indirect speech", and "the subject position in a clause" have all been identified. These structural features illuminate how the model's internal representations encode content along with the grammatical and discourse structure of the text it processes.

Perhaps most intriguing are features that appear to encode emotional and intentional states. Features activating for "sycophantic language", "the speaker expressing uncertainty", and "text from an AI assistant responding to a human" have been found. The latter category points toward a broader use of SAEs: understanding what the model knows and how it represents context, roles, and interaction patterns. This is potentially relevant to AI safety research, which aims to understand how models represent goals, intentions, and the relationship between themselves and their users.