Interpretability Goals › Approaches Overview

The chapters in Part LIX cover specific interpretability techniques. Here is a map of these methods, organized by what question each technique answers. Understanding this organization before diving into individual techniques helps you see how the field fits together and which methods address which goals.

Attention visualization (Chapter 2) extracts and plots the attention weight matrices from transformer models. These visualizations can reveal broad patterns: some heads specialize in attending to syntactic neighbors, others in long-range coreference, others in positional patterns, and a few in seemingly random patterns that defy easy interpretation. The technique is fast, requires no additional training, and produces visualizations that are intuitive to read.

The subsequent Chapter 3 on Attention Analysis Limitations is equally important: it documents why attention weights are not reliable indicators of causal importance. Understanding both the utility and the limitations of attention visualization is essential for interpreting the literature and applying the method without over-interpreting results. The appropriate use of attention visualization is as a hypothesis generator and a qualitative exploration tool, not as definitive evidence of causal importance.

Probing classifiers (Chapter 4) test whether a particular layer of a model has encoded a particular concept. The method trains a small linear classifier on the activations of a layer to predict some target property, such as part-of-speech tags, syntactic dependencies, or named entity types. If a simple classifier achieves high accuracy, that suggests the concept is linearly encoded in that layer's representation space, meaning the concept is accessible by a simple linear readout from the hidden state.

The simplicity of the classifier is an important design choice. If you use a highly nonlinear classifier, it might succeed by learning the concept itself rather than detecting whether the layer has encoded it. Using a linear classifier constrains the method to detecting only structure that is already explicit in the representation geometry.

Probing layers (Chapter 5) examines how encoding evolves across the depth of the network. For most linguistic properties, shallow layers encode surface features while deeper layers encode semantic and contextual properties. This layered structure is reproducible across model families and provides evidence about how language models build up representations progressively from token identities to discourse-level meaning.

The main limitation of probing is that it tests encoding, not usage. A layer may encode syntactic structure that the model learned incidentally without using it for the task at hand. Control experiments are needed to distinguish encoding from functional use, and researchers have developed several such controls, including comparing probing accuracy to that of a baseline that has access only to label frequency, and testing whether ablating the encoded structure changes model behavior.

Activation patching (Chapter 6) takes a causal approach. Instead of asking "what information is encoded here?" it asks "if I modify this activation, does the output change?" The method runs a model on two inputs, one "clean" and one "corrupted," and patches activations from the clean run into the corrupted run at specific sites. If patching a particular site restores the clean output, that site is causally implicated in the behavior. This causal framing addresses the faithfulness problem more directly than passive observation.

Activation patching has been used to locate where factual associations are stored in transformer networks, identifying specific MLP layers that act as "memory" for particular facts. It has also been used to trace where different linguistic computations happen, with results that broadly confirm the probing finding that syntax and semantics are encoded at different depths.

The logit lens (Chapter 7) is a complementary tool that tracks how predictions evolve across layers. By projecting intermediate activations back into the vocabulary space using the model's unembedding matrix, you can read off what the model is "predicting" at each layer, effectively asking: if the model had to generate a token right now, before seeing the remaining layers, what would it predict? This reveals when the final answer is committed to versus when the model is still processing and when it switches from one answer to another.

Sparse autoencoders (Chapter 8) address a fundamental challenge in neural network interpretation: individual neurons do not cleanly correspond to interpretable concepts. Research has found that neurons often respond to multiple unrelated features simultaneously, a phenomenon called polysemanticity. A single neuron might activate for both the concept "software developer" and the concept "baseball" with no obvious connection between them. This makes interpreting individual neurons unreliable.

Sparse autoencoders are trained on a layer's activations to find a dictionary of features, many more features than neurons, such that each activation can be expressed as a sparse combination of dictionary elements. The resulting features tend to be more interpretable than individual neurons because the sparse constraint forces each feature to be used for fewer distinct concepts. This is a form of dictionary learning applied to neural network activations.

Feature interpretation (Chapter 9) examines what these learned features represent. Automated methods analyze which inputs maximally activate each feature, looking for patterns in the text that explain the feature's behavior. Some features correspond to interpretable concepts (specific named entities, syntactic positions, stylistic properties, emotional tones), while others remain opaque. The proportion of interpretable features varies across layers and model sizes.

Mechanistic interpretability (Chapter 10) attempts to fully reverse-engineer the algorithms that specific components implement. Rather than asking what information is encoded where, it asks: what computation is this circuit performing, expressed as a precise algorithm? The classic example is the induction head circuit: a composition of two attention heads that implements an in-context pattern completion operation, enabling a model to continue a pattern like "A B ... A" with "B" by recognizing the previous occurrence of A and copying the token that followed it.

Mechanistic analysis has produced a small but growing catalog of circuits that implement specific capabilities in transformer models. The approach requires intensive manual work and careful experimental design, but it produces the most detailed understanding of any interpretability method. When mechanistic analysis succeeds, you end up with an algorithm written in terms of matrix operations that you can verify and simulate independently of the neural network.

Activation steering (Chapter 11) applies interpretability findings to influence model behavior. If you can identify a direction in activation space that corresponds to a concept (such as "honesty," "refusal," or "positive sentiment"), you can add a multiple of that direction to activations during generation to steer the model toward or away from that concept. This converts interpretability knowledge into a form of control, closing the loop between understanding and intervention. Activation steering has been used experimentally to suppress harmful outputs, enhance factual accuracy, and modulate emotional tone.