Sparse Autoencoders › SAE Architecture

A sparse autoencoder is a two-layer neural network with a bottleneck-free design. Unlike a standard autoencoder that compresses input into a smaller latent space (which forces the model to discard information), an SAE expands input into a much larger latent space. The goal is to find an overcomplete dictionary of features: a set of direction vectors larger than the input dimension, such that any activation vector can be approximately reconstructed as a sparse combination.

The overcomplete design enables interpretability. If the dictionary were the same size as the activation space ($m = d$), we would recover a standard basis, and the decomposition would simply express activations in terms of that basis without any interpretability gain. If the dictionary were smaller ($m < d$), we would be compressing the representation and losing information, which is the wrong direction. By making the dictionary much larger ($m \gg d$), we give the SAE enough capacity to find a basis where each concept corresponds to a distinct dictionary element, even though this requires representing the data as a sparse combination of many potential atoms.

Think of this as the difference between a compact codebook and a rich vocabulary. A compressed representation uses a small set of basis vectors to cover all possible inputs densely. A sparse overcomplete representation uses an enormous vocabulary where each word (feature) is highly specific, so most words are silent for any given input. The advantage of the latter is that silence is informative: if a feature for "legal Latin phrases" does not fire, you know the input is not legal Latin. Each feature carries meaning both when it fires and when it does not.

Given an activation vector $\mathbf{x} \in \mathbb{R}^d$ taken from a specific layer and token position in a language model, the SAE computes a two-step transformation.

Encoder:

The encoder subtracts a pre-bias, applies a linear transformation, and then zeros out negative values with ReLU. This produces a sparse, non-negative feature activation vector $\mathbf{z}$:

$$\mathbf{z} = \text{ReLU}(\mathbf{W}_{\text{enc}} (\mathbf{x} - \mathbf{b}_{\text{pre}}) + \mathbf{b}_{\text{enc}})$$

where:

• $\mathbf{x} \in \mathbb{R}^d$: the activation vector from the language model (input to the SAE)
• $\mathbf{b}_{\text{pre}} \in \mathbb{R}^d$: a pre-encoder bias that centers the activations before encoding
• $\mathbf{W}_{\text{enc}} \in \mathbb{R}^{m \times d}$: the encoder weight matrix mapping from model space to feature space
• $\mathbf{b}_{\text{enc}} \in \mathbb{R}^m$: the encoder bias, one entry per feature
• $\mathbf{z} \in \mathbb{R}^m$: the sparse feature activations (most entries are zero because of ReLU)
• $m \gg d$: the dictionary size is much larger than the model's hidden dimension, making the representation overcomplete

The pre-encoder bias $\mathbf{b}_{\text{pre}}$ deserves special attention. Activation vectors in language models tend to have a non-zero mean: there is a "baseline" activation pattern that most tokens share, corresponding to the model's general processing state regardless of token content. By subtracting this baseline before encoding, the SAE focuses on the token-specific deviations from the mean, which is where the interesting semantic content lives. Adding it back during decoding ensures the reconstruction is in the original activation space. This centering step is a small but meaningful improvement over naive SAE designs that omit it.

The ReLU nonlinearity is what creates sparsity. Any pre-activation below zero is clamped to exactly zero. Features with small positive pre-activations also tend toward zero because the $L_1$ penalty (discussed in the next section) pushes them there. The result is a feature vector where most entries are exactly zero and a handful have positive values. The number of non-zero entries is the effective $L_0$ of the representation, which is the primary sparsity metric used in practice.

Decoder:

The decoder reconstructs the original activation vector from the sparse feature activations using a linear combination of the dictionary columns:

$$\hat{\mathbf{x}} = \mathbf{W}_{\text{dec}} \mathbf{z} + \mathbf{b}_{\text{pre}}$$

where:

• $\mathbf{W}_{\text{dec}} \in \mathbb{R}^{d \times m}$: the decoder weight matrix whose columns are the dictionary features
• $\hat{\mathbf{x}} \in \mathbb{R}^d$: the reconstructed activation vector
• The pre-bias $\mathbf{b}_{\text{pre}}$ is added back to re-center the reconstruction in the original activation space

The decoder columns $\mathbf{W}_{\text{dec}}[:, j]$ are the dictionary features: unit-norm vectors in the model's activation space. When feature $j$ is active (i.e., $z_j > 0$), its dictionary vector contributes to the reconstruction proportionally to $z_j$. The reconstruction is a sparse linear combination: at most a few of the $m$ features are active, so the sum has few non-zero terms. The remaining $m - k$ features contribute nothing.

Notice that the encoder and decoder are separate, untied matrices. This is an important design choice. In a tied autoencoder, $\mathbf{W}_{\text{enc}} = \mathbf{W}_{\text{dec}}^T$, which enforces a symmetry that is convenient mathematically but is not necessary and may hurt performance. Untied weights allow the encoder to learn a geometry optimized for detecting features (high response to the feature direction, low response otherwise) and the decoder to learn a geometry optimized for reconstruction (accurate linear combination of feature vectors). In practice, the rows of $\mathbf{W}_{\text{enc}}$ and the columns of $\mathbf{W}_{\text{dec}}$ end up nearly parallel to each other after training, because pointing the encoder row in the direction of the corresponding decoder column is the most efficient detection strategy. But the freedom to diverge is occasionally useful.

The ratio $m / d$ is called the expansion factor. A typical SAE might use an expansion factor of 4x to 64x or more. For a model with $d = 4096$ hidden units, an expansion factor of 8 yields $m = 32768$ features. The higher the expansion factor, the more features are available, but training becomes more expensive and individual features may become less coherent.

There is a meaningful tradeoff here. With a small dictionary, the SAE must pack many concepts into few features, recreating the polysemanticity problem it was meant to solve. With a very large dictionary, the SAE may split a single concept into fine-grained fragments (e.g., "fruit" splitting into "tropical fruit", "citrus fruit", and "stone fruit") or learn redundant features that respond to nearly identical contexts. Feature splitting is not always undesirable: if you want to understand how the model distinguishes between varieties of fruit, you need the fine-grained split. But if you want a high-level view of what categories the model tracks, you want features at a coarser granularity. In practice, researchers often train a range of SAE sizes and evaluate feature quality both qualitatively (by inspecting which tokens cause each feature to activate) and quantitatively (by measuring reconstruction quality).

The expansion factor also interacts with the sparsity level. If you have 16x more features than dimensions, you need each activation to use roughly $1/16$ of the features or fewer to maintain meaningful sparsity. Very high expansion factors require correspondingly tight sparsity constraints to avoid the SAE becoming a dense lookup table. The relationship between expansion factor and required sparsity is approximately:

$$\text{expected interference} \approx \frac{k^2}{m}$$

where $k$ is the effective $L_0$ (number of active features). If $k = 10$ and $m = 160$ (expansion factor 10x for $d = 16$), the expected interference between active features is about 0.6, which is large enough to matter. If $m = 1600$, the interference drops to 0.06, making the superposition nearly negligible. This rough calculation explains why large expansion factors require tight sparsity: the more features you have, the fewer you can afford to use at once before interference degrades the representation.

A critical implementation detail is that the decoder columns are constrained to have unit norm after every gradient update. Without this constraint, the model could achieve sparse codes by making dictionary vectors very large and feature activations very small, avoiding sparsity in a trivial way. If the decoder column for feature $j$ has a large norm, then even a tiny activation $z_j = 0.001$ contributes substantially to the reconstruction, and the $L_1$ penalty on $z_j$ does not force it to zero because $0.001$ is already very small. The unit norm constraint removes this loophole by ensuring that achieving a strong reconstruction contribution for feature $j$ requires a large $z_j$, which the $L_1$ penalty can then force to zero.

After each parameter update step, we renormalize every decoder column to unit length:

$$\mathbf{W}_{\text{dec}}[:, j] \leftarrow \frac{\mathbf{W}_{\text{dec}}[:, j]}{\max(\|\mathbf{W}_{\text{dec}}[:, j]\|_2, 1)}$$

where:

• $\mathbf{W}_{\text{dec}}[:, j]$: the $j$-th column of the decoder matrix, representing feature $j$'s direction in activation space
• $\|\mathbf{W}_{\text{dec}}[:, j]\|_2$: the Euclidean norm of that column
• The $\max(\cdot, 1)$ prevents norm collapse for dead features while renormalizing columns that have grown beyond unit norm

This renormalization is applied for every feature $j \in \{1, \ldots, m\}$ after each gradient step. The constraint also has an interpretive benefit: because all decoder columns have the same norm, the magnitude of feature activations is directly comparable across features. A feature with $z_j = 3.0$ is "three times as active" as a feature with $z_k = 1.0$, in a geometrically meaningful sense.