Feature Interpretation › Code Implementation: Analyzing Feature Activations
Language AI HandbookAll chapters Feature Interpretation › Code Implementation: Analyzing Feature ActivationsOne section · Sections in this chapter6 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's build a practical toolkit for analyzing features from a trained SAE. We'll simulate the workflow using synthetic data, since running a real SAE requires substantial compute, but the analysis code translates directly to real SAE outputs.
Setup and Synthetic Feature Data
We'll create synthetic activation data that mimics the output of a real SAE: a sparse matrix where most values are zero, with a few positive activations per token.
import numpy as np
np.random.seed(42)
# Simulate SAE feature activations over a corpus
# n_tokens: number of token positions in the dataset
# n_features: number of SAE features
n_tokens = 10_000
n_features = 512
# Each token activates a small number of features (sparsity ~1%)
# Feature activation follows a power-law: some features are common, most are rare
feature_frequencies = np.random.zipf(a=1.8, size=n_features)
feature_frequencies = feature_frequencies / feature_frequencies.sum()
# Simulate which features activate for each token
activations = np.zeros((n_tokens, n_features), dtype=np.float32)
for token_idx in range(n_tokens):
# Sample which features fire (sparse: ~5-15 features per token)
n_active = np.random.randint(5, 15)
active_features = np.random.choice(
n_features, size=n_active, p=feature_frequencies, replace=False
)
# Activation strength drawn from a gamma distribution
activation_strengths = np.random.gamma(shape=2.0, scale=1.5, size=n_active)
activations[token_idx, active_features] = activation_strengthsActivation matrix shape: (10000, 512) Total non-zero activations: 94,859 Sparsity: 0.9815 (98.15% of entries are zero) Average active features per token: 9.5
The activation matrix is highly sparse. Most entries are zero, and each token position activates only a small fraction of all features. This sparsity is the defining characteristic of SAE representations and is what makes them tractable for interpretation. If every feature fired on every token, you could not identify any meaningful pattern from the activation profile alone. The sparsity is both a computational convenience and an epistemic tool: it makes the question "why did this feature fire here?" answerable rather than trivially true everywhere.
Computing Feature Statistics
# Compute per-feature statistics
activation_frequencies = (activations > 0).mean(
axis=0
) # Fraction of tokens that activate each feature
mean_active_value = np.where(activations > 0, activations, np.nan)
mean_active_values = np.nanmean(mean_active_value, axis=0)
max_active_values = activations.max(axis=0)
# Sort features by activation frequency
sorted_by_freq = np.argsort(activation_frequencies)[::-1]
# Identify "dead features" (never activate)
dead_features = np.sum(activation_frequencies == 0)Feature frequency stats: Min frequency: 0.00110 Max frequency: 0.97340 Mean frequency: 0.01853 Median frequency: 0.00290 Dead features (never activate): 0 Top-5 most frequent features (indices): [426 448 442 154 435] Top-5 least frequent features (indices): [143 274 201 341 328]
Most features have low activation frequency, consistent with the Zipf-like distribution we built in. The top features fire much more often than the median, and a handful of features may be "dead" (never activating), which is a known failure mode in SAE training. In a real SAE trained on a large corpus, the frequency distribution would span roughly six orders of magnitude: from features that fire on nearly every token (structural or syntactic catch-alls) to features that fire on fewer than one in a million tokens (highly specialized, rare concepts).
Activation Histogram Visualization
Finding Max-Activating Examples
In a real workflow you would look at actual text tokens; here we simulate by assigning each token a "concept label" drawn from a set of semantic categories. The key logic, however, is identical: sort by activation value, take the top , and inspect the associated metadata.
from collections import Counter
# Simulate semantic labels for each token (what concept each token represents)
concepts = [
"capital_city",
"financial_term",
"scientific_unit",
"emotion_word",
"discourse_marker",
"person_name",
"number",
"punctuation",
"verb_past",
"adjective",
]
# Assign concept labels to tokens
token_concepts = np.random.choice(
concepts,
size=n_tokens,
p=[0.05, 0.10, 0.07, 0.08, 0.06, 0.09, 0.15, 0.12, 0.13, 0.15],
)
def get_top_k_examples(feature_idx, k=20):
"""Return the top-k activating token indices and their activation values."""
feature_acts = activations[:, feature_idx]
top_k_indices = np.argsort(feature_acts)[::-1][:k]
top_k_values = feature_acts[top_k_indices]
active_mask = top_k_values > 0
return top_k_indices[active_mask], top_k_values[active_mask]
# Analyze a single feature
feature_to_analyze = sorted_by_freq[0] # Most frequent feature
top_indices, top_values = get_top_k_examples(feature_to_analyze, k=50)
top_concepts = [token_concepts[i] for i in top_indices]
concept_counts = Counter(top_concepts)Feature 426 -- Top-50 activating example concept distribution: number : 9 ######### adjective : 9 ######### verb_past : 7 ####### punctuation : 6 ###### person_name : 5 ##### capital_city : 4 #### financial_term : 4 #### discourse_marker : 3 ### emotion_word : 2 ## scientific_unit : 1 #
The concept distribution reveals which semantic categories dominate the feature's top-activating examples. If one concept accounts for 70% or more of the top examples, the feature is likely monosemantic with respect to that concept. If the examples split roughly equally, the feature may be polysemantic or tracking a more abstract property that cuts across several surface-level categories. The ASCII bar chart representation is a quick visual heuristic: a feature with one very long bar is a good candidate for a clean label; a feature with many roughly equal bars needs further investigation.
Measuring Monosemanticity via Entropy
The entropy score provides a quantitative summary of the concept-distribution analysis, allowing you to rank and filter features by their degree of monosemanticity without inspecting each one manually.
from scipy.stats import entropy
def compute_monosemanticity_score(feature_idx, k=50):
"""
Compute entropy-based monosemanticity: lower entropy = more monosemantic.
Returns (entropy_value, dominant_concept, dominant_fraction).
"""
top_indices, top_values = get_top_k_examples(feature_idx, k=k)
if len(top_indices) == 0:
return None, None, 0.0
top_concepts_local = [token_concepts[i] for i in top_indices]
counts = Counter(top_concepts_local)
total = sum(counts.values())
probabilities = np.array([c / total for c in counts.values()])
h = entropy(probabilities, base=2)
dominant = counts.most_common(1)[0]
return h, dominant[0], dominant[1] / total
# Analyze monosemanticity for a sample of features
sample_features = sorted_by_freq[:100] # Top-100 by frequency
entropy_scores = []
dominant_concepts = []
dominant_fractions = []
for feat_idx in sample_features:
h, dom_concept, dom_frac = compute_monosemanticity_score(feat_idx, k=50)
if h is not None:
entropy_scores.append(h)
dominant_concepts.append(dom_concept)
dominant_fractions.append(dom_frac)
entropy_scores = np.array(entropy_scores)
dominant_fractions = np.array(dominant_fractions)Monosemanticity analysis (top-100 features by frequency): Mean entropy: 3.096 bits Median entropy: 3.106 bits Features with entropy < 1.5 bits (near-monosemantic): 0 Features with entropy > 2.5 bits (polysemantic): 100 Mean dominant concept fraction: 0.195
Features with entropy below 1.5 bits have a dominant concept covering at least 70% of their top examples; these are candidates for clean, monosemantic labels. Features above 2.5 bits distribute broadly across categories and require more careful investigation to determine whether they are polysemantic or whether the concept categories are too coarse for the feature's actual function. The mean dominant concept fraction gives another view of the same information: when this value is close to 1.0, most features are near-monosemantic; when it is close to , features are broadly polysemantic.
Monosemanticity Scatter Plot
A monosemantic feature appears in the lower region of this plot: low entropy and green coloring indicate that a dominant concept covers most of the top- examples. Polysemantic features cluster in the upper portion. The relationship between frequency and monosemanticity is not tight in this synthetic data; in real SAEs, frequent features often turn out to be polysemantic (they fire broadly because they track a structural property shared by many contexts), while rare features can be either highly specific (monosemantic) or underspecified (poorly trained).
The side-by-side comparison illustrates the difference in activation profiles. The monosemantic feature (left) shows a narrow, peaked distribution: when it fires, it fires with a consistent strength, because it is responding to a single, well-defined concept. The polysemantic feature (right) shows a broader, flatter distribution: its activations are more variable because it is responding to multiple distinct contexts, each of which may activate it to a different degree.