Sparse Autoencoders › Worked Example: SAE on Simple Activations

To build intuition, let's work through how an SAE would learn to decompose a small synthetic dataset of activations. Imagine we have a 2-dimensional activation space, but the actual generative factors are 4 binary features: $f_1$ (indicates a verb), $f_2$ (indicates a proper noun), $f_3$ (indicates negation), $f_4$ (indicates past tense). Each activation vector is a linear combination of whichever features are active, with some noise added.

The SAE should learn to recover these 4 underlying features from the 2D activations, even though the problem is technically underdetermined (we have more features than dimensions). This is possible because of sparsity: at most one or two features are active at a time, which makes the problem well-conditioned. Intuitively, if you observe a large collection of 2D vectors and know that each was generated by at most 2 of 4 possible patterns, the geometry of the data (clustering near the sparse combinations) reveals the underlying patterns.

Let us trace through this concretely. Suppose feature $f_1$ (verb) corresponds to the direction $(1, 0)$ and feature $f_2$ (proper noun) corresponds to the direction $(-0.5, 0.87)$ in 2D space. When we observe an activation vector near $(0.9, 0.1)$, sparsity tells us it is most likely a single-feature activation for $f_1$. When we observe a vector near $(-0.4, 0.75)$, it is likely a single-feature activation for $f_2$. When we observe a vector near $(0.5, 0.9)$, it could be a two-feature combination of $f_1$ and $f_2$. The SAE encoder learns to map each of these observations to the correct sparse code by projecting the input onto the feature directions and thresholding.

The key insight is that you cannot determine this decomposition from any single observation; you need the statistics of many observations. The SAE essentially learns that the data is concentrated near a small set of directions, with some observations at intermediate points from sparse combinations. This geometric structure is what the ReLU encoder exploits. Directions with high dot product against the input are likely active features, and the decoder learns to reconstruct the input as a combination of those directions.

In practice, the dictionary directions for the 4 features must be nearly orthogonal to allow unique decoding. With 4 features in 2D, they cannot be fully orthogonal (orthogonality requires no more features than dimensions), but the near-orthogonality of high-dimensional space makes this tractable at the scale of real language models. A 4096-dimensional residual stream can comfortably support tens of thousands of nearly orthogonal features, and the near-orthogonality is what ensures each feature can be decoded independently.

This also explains why the SAE approach breaks down in very low-dimensional spaces. The 2D example above is illustrative but not realistic: with only 2 dimensions and 4 features, the features are forced to interfere substantially. As dimension grows, the interference per feature pair decreases as $1/d$, making the decomposition increasingly reliable. Language models, with hidden dimensions in the thousands, operate in a regime where the interference is small enough to make SAE decompositions informative.