Interpretability Goals › The Relationship Between Interpretability and Model Development

Interpretability research does not exist in isolation from model development. The two activities are in dialogue in ways that are worth understanding.

On one side, model development informs interpretability: as models evolve (larger, more capable, trained on more data), interpretability researchers need to adapt their methods to remain applicable. Techniques that work on BERT may not scale to GPT-4-sized models. Circuit analyses that fully characterize a capability in GPT-2 may find that the same capability is distributed across far more components in a larger model, making full characterization infeasible.

On the other side, interpretability informs model development in increasingly concrete ways. Understanding which layers store factual knowledge (via activation patching) has informed the design of knowledge editing methods that modify those specific layers. Understanding how attention heads specialize has informed efforts to prune models by removing heads that are not useful for a particular downstream task. Understanding polysemanticity has motivated research into architectures that encourage more interpretable, monosemantic representations.

This feedback loop suggests that interpretability can guide model development as well as retrospective analysis. It can inform training choices, architectural choices, and evaluation standards. Models could in principle be designed with interpretability as a constraint, rather than treating it as an afterthought. This direction, sometimes called "inherently interpretable" or "transparency-by-design," is an active research area that trades some raw capability for increased explainability.