Sparse Autoencoders › Evaluating SAE Quality

Evaluating SAEs is an active research area because there is no single ground-truth measure of feature quality. Researchers use a combination of quantitative and qualitative metrics, and the best-performing SAEs by one metric are not always the best by others. This makes SAE evaluation difficult.

The core quantitative tradeoff is between reconstruction fidelity and sparsity:

• Variance explained ($R^2$): What fraction of the variance in activation vectors is explained by the SAE? Higher is better. This is the most commonly reported metric and is easy to compute, but it does not distinguish between faithful geometric reconstruction and reconstruction of task-relevant information.
• Mean $L_0$: The average number of features active per token. Lower means sparser. Combined with $R^2$, this defines the Pareto frontier used to compare SAEs at the same sparsity level.
• Loss recovered: When you replace language model activations with SAE reconstructions and run the model forward, how much does the language model's loss increase? A loss recovered of 100% means no degradation. This metric is more meaningful than $R^2$ for practical interpretability work because it captures whether the SAE preserves the information the model uses for predictions, rather than variance that may be irrelevant noise.
• CE loss difference: The absolute increase in cross-entropy loss when the model uses SAE-reconstructed activations. Smaller is better. A model with good $R^2$ but high CE loss difference is geometrically accurate but discarding task-relevant information.

A useful summary is the Pareto frontier of reconstruction quality vs. sparsity across different values of $\lambda$. A better SAE traces a frontier further from the origin: for any given sparsity level, it achieves better reconstruction.

Beyond numerical metrics, researchers evaluate SAEs qualitatively. These evaluations require human judgment or proxy language models:

• Monosemanticity rate: What fraction of features, when inspected, have a clear, coherent semantic interpretation? This is typically measured by having human annotators or automated language model labelers review the top-activating examples for each feature.
• Feature coherence: Do the top-activating examples for a feature share a specific semantic property, or do they span unrelated concepts? A feature is coherent if you could describe it with a single short phrase and have someone use that description to predict (with above-chance accuracy) whether a new example would activate the feature.
• Recall of known circuits: When you use the SAE to analyze known circuits (e.g., induction heads, factual recall mechanisms), do the SAE features correspond to the components identified by other interpretability methods? This is a cross-validation of the SAE against independent evidence.
• Activation steering consistency: When a feature is artificially activated via steering, do the resulting model outputs match the expected semantic content of the feature? A feature labeled "Paris" should, when steered up, increase the probability of Paris-related tokens.

The last two metrics are particularly important because they connect the SAE's features to causal claims about model behavior. A feature might be statistically coherent (high monosemanticity rate) but causally inert: it activates for a concept but does not causally influence what the model outputs. For interpretability research, causal relevance is what ultimately matters.