Sparse Autoencoders › Key Parameters

Successful SAE training requires careful attention to several key parameters. The choices interact with each other, so understanding the tradeoffs is important:

• d_sae (dictionary size): The number of features in the dictionary. Larger dictionaries find more fine-grained features at greater computational cost. Typical expansion factors range from 4x to 64x. Too small risks polysemantic features; too large risks feature splitting and increased dead features.
• lambda_l1 (sparsity coefficient): Controls the tradeoff between reconstruction quality and sparsity. Larger values produce sparser but less accurate reconstructions. Must be tuned carefully; typical values range from $10^{-4}$ to $10^{-1}$. The right value depends on the expansion factor and the intrinsic sparsity of the activation distribution.
• hookpoint: Which component of the language model to extract activations from. Residual stream activations are the most common choice for general-purpose interpretability. MLP outputs work better for studying factual knowledge storage.
• n_training_tokens: The number of token activations to train on. High-quality SAEs typically require billions of tokens to cover rare features. The requirement scales with the rarity of the rarest features you want to capture.
• activation function: ReLU with $L_1$ penalty, TopK, or JumpReLU. TopK is increasingly preferred for its stability and elimination of the dead feature problem. JumpReLU offers per-feature threshold learning at some additional complexity.
• decoder normalization: Always normalize decoder columns to unit norm after each gradient step. Omitting this allows the model to trivially satisfy the sparsity constraint without learning sparse representations.