Feature Interpretation › Limitations and Practical Considerations

Feature interpretation is a young and rapidly evolving field. The methods described in this chapter are already producing remarkable results, but several significant challenges limit the conclusions that can currently be drawn from SAE analyses.

Automated interpretability pipelines produce natural language descriptions, but natural language is imprecise and context-dependent in ways that can mislead. A feature described as "mentions emotions" may instead be tracking a subtle distributional property that the description inadequately captures: perhaps it fires specifically on first-person expressions of negative affect in formal writing, a pattern that "mentions emotions" fails to represent. The autointerpretability score measures correlation between a description-based simulator and actual activations, but correlation is not causation or completeness.

A description could score well on the simulation task but still be fundamentally wrong if the proxy task is too easy, or if the description happens to be coextensive with the true pattern even though the concepts are unrelated. Conversely, a correct but complex description might score poorly because the simulator model cannot reliably implement the described rule in its one-shot simulation task.

The deeper problem is that there is no ground truth for feature interpretation. Unlike in supervised learning, where the correct answer is defined by a label, the "correct" interpretation of a feature direction in activation space is not formally defined. Features encode statistical patterns in the training data, and those patterns do not always have clean, human-interpretable labels. The best we can say is that a description is more or less useful, more or less predictive, and more or less consistent with the available evidence.

Even if every individual feature is well-interpreted, understanding model behavior still requires understanding which features are active for a given input, how they interact across layers, and what the model ultimately concludes. Feature circuits partially address this, but tracing a complete circuit through all layers of a modern large language model with dozens of layers and billions of parameters is computationally demanding. The resulting circuits can be complex enough to resist easy interpretation even after all the nodes and edges have been identified.

There is also a compositionality question: many model behaviors emerge from the interaction of hundreds or thousands of features, not from a small, clean circuit. The transformer's attention mechanism can create complex, input-dependent interactions between distant token positions. When the functional circuit for a behavior involves hundreds of features interacting across twenty layers, circuit analysis may reveal all the components without providing a satisfying explanation of how they work together. The circuit is correct but not comprehensible.

This problem is sometimes called the "legibility gap": the description of a circuit may be technically accurate (it correctly identifies all the relevant features and edges) while still being too complex for a human to understand. Bridging the legibility gap may require finding higher-level abstractions that summarize circuit structure, analogous to how a software engineer understands a program in terms of functions and modules rather than individual machine instructions.

A well-trained SAE should recover monosemantic features, but in practice SAE training involves tradeoffs that compromise this ideal. The sparsity penalty ($L_1$ regularization on activations) pushes features toward monosemanticity by encouraging each feature to fire only when a specific concept is present. But the sparsity penalty also introduces a systematic bias: it causes the SAE to underestimate activation values, because the optimizer knows it will be penalized for any positive activation and therefore sets the threshold conservatively.

When many features are active simultaneously (as happens for complex, information-rich tokens), the SAE must choose how to distribute the representation across available features. This distribution is shaped by the features that best match the input and by the interaction between the sparsity penalty and the reconstruction objective. The result is that some features are shaped by their primary concept and by their secondary role in reducing reconstruction error for other inputs. The feature directions are not purely monosemantic but are subtly distorted by the learning process.

Detecting and correcting for this distortion is an active research problem. One diagnostic approach is to train multiple SAEs with different dictionary sizes and sparsity levels on the same model, and check whether the identified feature directions are stable across training runs. Stable directions are more likely to represent stable structure in the model's representations; unstable directions may be artifacts of a particular SAE training configuration.

The computational cost of interpreting every feature scales with both model size and dataset size. Collecting top-$k$ activating examples for millions of features requires running forward passes over a corpus large enough to see meaningful statistics for rare features. For a feature that fires on one in a million tokens, you need at least tens of millions of tokens just to gather a reliable top-$k$ set, and potentially billions of tokens to see the full distribution of contexts where the feature fires.

This creates a fundamental resource inequality. Feature interpretation at the quality described in research papers from Anthropic and other major labs requires infrastructure at roughly the scale of the original model training: the same hardware, similar compute costs, and large engineering teams. Smaller research groups and individual practitioners cannot conduct this kind of exhaustive analysis for the models they work with.

There is an ongoing effort to democratize feature interpretation. Projects like Neuronpedia provide public dashboards for exploring SAE features in open-source models, allowing practitioners to browse top-activating examples and read automated descriptions without running large-scale inference themselves. The EleutherAI and other open research groups have released SAE weights trained on open-source models like Pythia and LLaMA, enabling the community to conduct feature analysis without training from scratch. These resources lower the barrier to entry significantly, though they do not resolve the fundamental scaling problem for frontier models.

A recurring challenge in feature interpretation is the absence of a reliable evaluation framework. It is relatively straightforward to produce a feature label and much harder to verify that the label is correct and complete. Current evaluation methods include:

• Held-out simulation: testing whether the description predicts activations on new examples not used to generate the description. This measures precision but not recall: a description that fires on exactly the right tokens but also misses many tokens where the feature fires will score well on precision but poorly on recall.
• Ablation studies: removing or suppressing the feature's activation and checking whether the model's behavior changes as predicted. This tests causal relevance but requires knowing which behavior to monitor, which is itself a non-trivial interpretability question.
• Steering experiments: amplifying the feature and checking whether the model's outputs shift in the expected direction. This provides a complementary causal test to ablation, but steering may produce side effects if the feature direction is not perfectly isolated from other features.
• Human agreement: having multiple researchers independently label the same features and measuring inter-annotator agreement. This tests labeling consistency but not accuracy: two researchers can agree on an incorrect label.
• Cross-model comparison: checking whether the same feature direction (or a close neighbor) appears in independently trained models. Convergent features across training runs are stronger candidates for representing semantic content rather than artifacts of a particular training seed.

None of these methods is fully satisfactory in isolation. A feature that passes all five tests is a strong candidate for a correct interpretation, but even then the interpretation may be incomplete (covering the primary concept but missing secondary roles), imprecise (using a coarser category than the true concept), or wrong in rare edge cases. Building a rigorous evaluation standard for feature interpretation is an active research area, closely tied to the broader question of what "understanding a model" should mean.