Interpretability Goals › Scope Levels

Interpretability can be applied at different levels of scope, and the appropriate level depends on the question you are trying to answer. Choosing the right granularity is not trivial: going too coarse (model-level analysis) can obscure the mechanisms you want to understand, while going too fine (neuron-level) can produce explanations too detailed to be actionable. Most productive interpretability studies operate at two or three scales simultaneously, using coarse analysis to identify where to look and fine-grained analysis to explain what is found there.

Token level analysis examines the relationship between specific input tokens and output tokens. Feature attribution methods operate at this level, assigning scores to input tokens that indicate their contribution to a particular output. Token-level analysis is the most common form of explanation presented to end users, because it is intuitive: highlighting words in the input that influenced the answer resembles how humans explain their reasoning. The limitation is that token-level attribution does not explain how the model processes those tokens internally. Knowing that the word "Paris" was important to the model's output of "France" tells you that the model is using that information, but not whether it is doing so through a learned association, a factual retrieval, or a chain of intermediate computations.

Layer level analysis examines how representations change as information flows through the depth of the network. Probing classifiers and the logit lens operate at this level, revealing how different properties emerge at different depths. Layer-level analysis has produced some of the most consistent findings in interpretability research: syntactic structure tends to peak in early-to-middle layers, while semantic and world-knowledge properties peak in later layers. These patterns replicate across model families (BERT, GPT, T5), suggesting they reflect something fundamental about how depth interacts with representation quality. The finding is robust enough that it now informs architectural choices: tasks that require syntactic processing can sometimes be handled with smaller models by accessing representations at shallower layers.

Head level analysis examines specific attention heads within a layer. Individual heads tend to specialize in particular patterns (syntactic, positional, semantic, copying), and identifying these specializations is a key step in mechanistic analysis. Early work by Clark et al. (2019) showed that specific heads in BERT reliably tracked syntactic dependencies, direct objects, and coreference relations. Ablating individual heads can test whether they are necessary for a behavior or merely correlated with it, which is an important control for distinguishing causal roles from incidental correlations.

Circuit level analysis examines compositions of multiple components. Induction heads, for example, are a circuit involving two heads across two layers, and understanding the circuit requires understanding how the first head sets up a pattern that the second head completes. Circuit analysis is the most detailed form of interpretability but also the most labor-intensive. It typically combines activation patching, attention head ablations, and manual inspection to reverse-engineer what computation a group of components is collectively performing. The payoff is that circuit-level understanding produces algorithmic explanations, not just correlational ones.

Model level analysis examines global properties of the model, such as overall representation geometry, distribution of factual knowledge across layers, or high-level behavioral properties. Model-level analysis is often the entry point for a new investigation: it tells you where in the model to look before you zoom in. Centered kernel alignment (CKA) similarity matrices computed across all layer pairs, for instance, can reveal whether a model has a distinct "processing phase" in early layers and a "retrieval phase" in later layers before you have examined any individual component. Model-level results are less mechanistically informative than circuit-level results, but they are much faster to obtain and can efficiently narrow the search space.