Mechanistic Interpretability › Limitations and Challenges

Mechanistic interpretability is a young and rapidly evolving field. Several significant challenges constrain what current methods can achieve, and understanding these challenges is important for evaluating what circuit-level findings establish.

Scale. The circuits discovered so far are primarily in small models (GPT-2 scale, two-layer models). Modern large language models have hundreds of layers and tens of thousands of attention heads. Manually tracing circuits at this scale is not feasible. Automated circuit discovery methods (using systematic activation patching at scale) are being developed, but they remain computationally expensive and produce results that are harder to verify. The tools developed for small models do not trivially generalize: the residual stream in a 70-billion-parameter model carries far more information than in a two-layer toy model, and the circuits are correspondingly more complex and intertwined.

Polysemanticity. Even with SAE features as the unit of analysis, circuits may be messy because features interact in complex, context-dependent ways. The "same" circuit may implement slightly different computations in different contexts, making clean characterizations difficult. A head that acts as a copy circuit for token identity may behave differently when the token appears in a specific syntactic context. These context-dependencies can make circuit descriptions that are accurate on average misleading for specific edge cases.

Non-linear interactions. The residual stream decomposition and the linear representation hypothesis assume that contributions add approximately linearly. But MLP layers implement nonlinear functions, and interactions between components can create effects that are not captured by studying components in isolation. Activation patching mitigates this by being causally rather than linearly oriented, but the full picture of how nonlinear interactions contribute to circuit behavior remains incomplete. When two components interact nonlinearly, the effect of patching one component depends on the state of the other, and isolating individual contributions becomes ill-defined.

Completeness vs. comprehensibility. A circuit that truly accounts for all the causal paths contributing to a behavior may be very large and complex, defeating the goal of comprehensibility. Researchers must make choices about which components to include in a circuit description, which introduces a selection bias. Different teams may find different "circuits" for the same behavior depending on their methodology, their choice of importance threshold, and which components they chose to analyze. This makes it difficult to compare circuit analyses across papers and to assess how complete any given analysis is.

Training vs. final model analysis. Most mechanistic interpretability work analyzes fully trained models. But the circuits found in a model depend on the training data, training procedure, and hyperparameters. Circuits may not be stable across models trained with the same architecture but different random seeds or data orders. This limits the generality of individual circuit findings. A circuit found in one GPT-2 instance may not appear in identical form in another, raising questions about whether circuit analysis reveals architectural truths or training-run idiosyncrasies.

Despite these limitations, mechanistic interpretability has produced concrete scientific results. The discovery of induction heads, the characterization of name mover circuits, and the identification of specific algorithmic implementations (Fourier circuits for modular arithmetic, greater-than circuits) have provided principled, verifiable accounts of specific model behaviors. These results would have been impossible under a purely behavioral view of neural networks. Each verified circuit finding advances the field's capacity to understand and ultimately predict model behavior at a computational level.

The field is also developing better tooling rapidly. Libraries like TransformerLens make it easy to apply activation patching and weight analysis to GPT-2-scale models. Automated circuit discovery methods are extending the reach of the methodology to larger models. And the integration of SAE-based feature analysis with circuit analysis is creating a richer vocabulary for describing model computations. The path forward requires both better tools and better theory, and both are actively developing.