Interpretability Goals › Limitations of Interpretability Research

Interpretability is a young field with substantial unsolved problems. Being clear about current limitations matters for setting realistic expectations and for identifying where the field most needs to advance.

Scalability. Most detailed interpretability work has been done on small models: GPT-2 (1.5B parameters), BERT-base (110M parameters), and smaller. The computational costs of mechanistic analysis grow with model size, and it is unclear how well techniques that work for small models will scale to frontier models with hundreds of billions of parameters. A circuit that is implemented by five attention heads in a small model might be distributed across hundreds of heads in a larger model, making characterization vastly harder.

Incompleteness. No existing method gives a complete account of a model's computations. Probing tells you about encoding but not usage. Attribution methods identify important inputs but not why they are important. Mechanistic analysis has fully characterized only a handful of circuits in any model, and these circuits account for only a tiny fraction of the model's total computational capacity. The gap between what we can explain and the total behavior of a frontier model is enormous.

Faithfulness verification. Verifying that an explanation is faithful is itself a hard problem. The most rigorous check is causal: if you modify the thing the explanation says is responsible, does the output change as predicted? But this verification is computationally expensive, requires careful experimental design, and may not always be feasible for complex behaviors. For behaviors that emerge from many components jointly, ablating any individual component may not produce the predicted change, even if that component contributes.

Evaluation standards. Unlike supervised learning, interpretability lacks standardized benchmarks. Different papers use different criteria for what counts as a successful explanation, making it hard to compare methods or measure progress objectively. A paper that claims to have "identified the circuit responsible for indirect object identification" may be making a much stronger or weaker claim than it appears, depending on what precisely "circuit" and "responsible" mean in that context.

Polysemanticity. Individual neurons and features in large models tend to respond to many unrelated concepts simultaneously. This makes it harder to assign clean semantic interpretations to model components. Sparse autoencoders (Chapter 8) address this partially, but the underlying polysemanticity reflects fundamental properties of how neural networks use their capacity efficiently. A neural network trained by gradient descent will tend to pack as much information as possible into its available neurons, and superposition of many features onto few neurons is an efficient way to do this.

Reverse engineering vs. prediction. Interpretability aims to understand how a model works, but understanding may not be sufficient to predict or prevent failures. A model that is fully understood in principle might still fail in ways that surprise you in practice, because the space of possible inputs is too large to reason about exhaustively. Understanding the mechanism does not automatically give you the ability to predict all outputs from all inputs.

The gap between description and explanation. Many interpretability findings are more accurately described as descriptions than explanations. Finding that a particular attention head attends to the previous token describes a pattern; explaining why that pattern produces the observed model behaviors requires connecting the description to a causal account of the computations. The transition from description to explanation requires additional experimental work that is often not fully carried out.

Despite these limitations, interpretability research has already produced actionable findings: identification of factual storage sites in MLP layers, characterization of induction head circuits, development of sparse autoencoders that find interpretable features, and causal tracing tools that locate where specific computations occur. The field is making measurable progress, even if the goal of fully understanding large language models remains distant.