Sparse Autoencoders › The Sparse Coding Hypothesis

The motivation for sparse autoencoders comes from a long history in both computational neuroscience and signal processing. In 1996, Olshausen and Field showed that simple cells in the visual cortex can be understood as implementing a sparse code over natural images. When you train a dictionary on natural image patches and force sparse reconstruction, the learned atoms look like oriented Gabor wavelets, which is exactly what is found in primary visual cortex. The brain appears to use sparsity as an efficiency principle: by representing each stimulus with a small number of highly selective neurons, the brain minimizes metabolic cost while maximizing information capacity.

This insight from neuroscience is not coincidental. Efficient coding theory predicts that a system designed to transmit information under metabolic constraints will use a sparse code. If neurons are expensive to fire (and in the brain, they are), then the optimal strategy is to allocate firing to a small number of neurons that carry highly specific, non-redundant information. The rest remain silent. A neuron that fires for almost everything conveys little information per spike. A neuron that fires only when a very specific pattern is present conveys a lot.

The same insight has been applied to language model representations. Anthropic researchers, building on earlier work by Cunningham et al. (2023), demonstrated that the activation space of transformer language models can be decomposed into a large dictionary of sparse features, many of which are monosemantic: each feature fires for a coherent semantic concept (e.g., "names of fruits", "past tense verb forms", "code comments"). This result was surprising because the raw neuron activations showed severe polysemanticity, and it validated the core hypothesis that language models, like biological brains, may use sparse distributed codes.

The key intuition is that a model may have been trained with far more conceptual features in the world than it has neurons to represent them. The world has millions of meaningful concepts: countries, emotions, grammatical categories, coding patterns, historical events, and so on. A language model with tens of thousands of hidden units per layer cannot allocate a dedicated neuron to each concept. The network solves this problem by superposing many features across the same set of neurons, with each feature using a different linear direction in activation space. When features are superposed in this way, they look polysemantic when you examine individual neurons, but a sparse reconstruction algorithm can in principle separate them.

This is possible because high-dimensional spaces are roomy. In two dimensions, you can fit at most two perfectly orthogonal vectors. In a thousand-dimensional space, you can fit millions of vectors that are nearly orthogonal (with inner products close to zero). The language model exploits this geometric freedom by storing features as "almost orthogonal" directions, accepting small amounts of interference between features in exchange for enormous representational capacity.

Consider a concrete example of how superposition creates polysemanticity. Suppose a model has a 512-dimensional residual stream but needs to represent 10,000 semantic features. It cannot give each feature its own dimension. Instead, it assigns each feature a unit vector in that 512-dimensional space, chosen to be nearly orthogonal to all other features. When a feature fires, it adds its unit vector (scaled by the activation magnitude) to the residual stream. When multiple features fire simultaneously, their vectors add up. If no two features fire together very often (sparsity), the sum almost always contains just one or two terms, making it relatively easy to decode which features contributed. If features fire densely (nearly all the time), the sums are hopelessly entangled and decoding becomes impossible.

This is why sparsity is essential to the whole enterprise. The sparse coding hypothesis is not just "representations are sparse for efficiency." It is "representations must be sparse for the superposition to remain decodable." Without sparsity, a language model storing 10,000 features in 512 dimensions would produce activation vectors that are linear superpositions of hundreds of features simultaneously, and no algorithm could reliably disentangle them. Sparsity is what makes the geometry work.

The following visualization shows the core idea of superposition in two dimensions. Three features are packed into two-dimensional space by using directions that are not orthogonal. Each feature activates rarely (sparse), so the interference between them is low on average, even though they share the same space.