Interpretability Goals: Debugging, Trust, and Safety

When a language model gives a wrong answer, the obvious question is: why? When a model refuses a reasonable request or agrees with a harmful one, the same question arises. When a deployed system produces confident-sounding text that is factually incorrect, you need more than a retry strategy. The broader field of AI safety, covered in Part LX, provides defenses at the system boundary: guardrails, content filters, red teaming. But these approaches treat the model as a black box. Interpretability research asks something more fundamental: what happens inside?

Interpretability is the study of how neural networks work internally. It tries to answer questions like: which parts of the model are responsible for a given output? What representations does the model build internally? Does the model "know" facts it sometimes gets wrong? Can we predict when the model will fail? Do the internal computations correspond to anything semantically meaningful from a human perspective? These questions matter for practical reasons and for deeper scientific ones, and the gap between asking them and answering them has driven one of the most active research agendas in modern deep learning.

This chapter surveys the goals and approaches of interpretability research before the later chapters in Part LIX examine specific techniques. You will learn why interpretability is pursued, what problems it solves, and how researchers currently approach it. Understanding the goals helps you evaluate any specific technique: not just "does this method work?" but "work for what purpose?" Different stakeholders want different things from interpretability, and a technique that perfectly satisfies one goal may be almost useless for another.

The term interpretability is sometimes used interchangeably with explainability, but researchers in the field have started to distinguish the two. Explainability often refers to post-hoc methods that produce a human-legible explanation for a specific output, such as a saliency map highlighting important input tokens. Interpretability, in the stronger sense used in mechanistic interpretability research, refers to understanding the model's internal algorithm, not just describing its outputs. A model can be highly explainable (every output comes with a confidence score and highlighted tokens) while remaining deeply uninterpretable (you still have no idea what computation produced that output). Keeping this distinction in mind will help you evaluate claims in the literature more critically.

The case for interpretability is not one argument but several. Different stakeholders care about interpretability for different reasons, and no single technique satisfies all of them. Before examining any technical method, it is worth being precise about what you are trying to accomplish. The four most important motivations are debugging and error analysis, trust and accountability, safety and alignment, and scientific understanding. Each motivates a somewhat different set of methods and a different definition of what counts as a successful explanation.

The most immediate motivation for interpretability is practical: models make mistakes, and understanding why they fail helps you fix them.

Training a language model is a high-dimensional optimization process that minimizes a loss function over a dataset. But a lower training loss does not guarantee that the model has learned the intended patterns. The model might have learned spurious correlations in the data, such as associating a formatting style with a correct answer rather than understanding the underlying facts. It might perform well on the test set while failing on inputs that look superficially different. It might have memorized specific training examples rather than generalizing principles. Without interpretability tools, diagnosing these failures is essentially guesswork, and fixing them is even harder.

Consider a named entity recognition model that performs well on news articles but fails on social media text. Without examining what features drive its predictions, you cannot know whether the failure is due to abbreviations, informal capitalization, different entity types, out-of-vocabulary terms, or something else entirely. With probing tools (covered later in this part), you can examine which layers encode entity-type information and whether that encoding degrades on social media inputs. With attribution methods, you can identify which specific tokens or features are driving incorrect predictions. That diagnostic information points you toward a fix: perhaps adding social media text to training, or normalizing abbreviations at preprocessing time.

The same logic applies to more serious failure modes. If a language model consistently produces incorrect answers about a particular domain, interpretability tools can reveal whether the problem is that the relevant knowledge was never encoded, that it was encoded but in the wrong layer, that it is overshadowed by competing associations, or that the decoding process is retrieving the wrong representation. Each diagnosis suggests a different remedy.

Debugging benefits from interpretability at multiple granularities:

• Instance-level explanations show why a specific prediction was made, which is useful for examining individual failures and understanding edge cases.
• Aggregate analysis reveals systematic biases and common failure patterns across a dataset, identifying classes of inputs the model handles poorly.
• Component-level analysis identifies which attention heads, layers, or circuits contribute to specific capabilities or errors, enabling targeted interventions.

There is also an important asymmetry between debugging and production monitoring. When a model fails in deployment, you often have only the input and output, with no access to intermediate states. Interpretability tools that can diagnose failures from external behavior alone (like attribution methods that only need input-output pairs) are particularly valuable in this setting. Tools that require access to activations require re-running the model in a controlled environment, which is more expensive but produces more informative results.

Deploying a language model in a high-stakes context, such as medical information retrieval, legal document analysis, financial advice, or hiring support, raises questions about accountability. When a system makes a decision that affects someone, the people affected (and the people deploying the system) need some basis for evaluating whether the decision was reasonable.

This concern extends beyond technical practice. Regulatory frameworks in many jurisdictions require explanations for automated decisions. The EU AI Act imposes transparency requirements on high-risk AI systems. Various national-level frameworks for automated decision-making (including GDPR's "right to explanation" provisions) require that organizations be able to explain algorithmic decisions to affected individuals. Even absent legal requirements, users tend to trust systems more when they understand the basis for outputs. A medical professional who can see which parts of a patient record the system found most relevant is in a better position to check the system's reasoning than one who receives only a recommendation.

Interpretability research addresses trust in two related ways. First, it enables after-the-fact explanations: given an output, you can construct an explanation of which inputs or internal states contributed to it. Second, it enables proactive audit: you can examine a model's internal representations to check for undesirable patterns before deployment, rather than discovering problems in production. Both uses are valuable, but the second is arguably more powerful, because it can prevent harms rather than explain them retroactively.

The challenge is that trust built on explanations is only valuable if those explanations are accurate. A plausible-sounding explanation for a wrong answer is worse than no explanation, because it creates false confidence. Imagine a system that explains its loan rejection by citing an applicant's income instability, when in fact the driving factor was a feature proxy for ethnicity. The explanation provides a veneer of legitimacy without revealing the true basis for the decision. This motivates a critical distinction: explanations need to be faithful (reflecting the model's internal process) rather than merely plausible (consistent with the output but not necessarily the cause). We will return to this distinction throughout the chapter.

A related challenge is the expertise gap. Explanations are only useful if the recipient can understand and act on them. A visualization of attention head activations may be informative to an NLP researcher but opaque to the physician, judge, or loan officer who needs to evaluate the system's decision. Interpretability research needs to consider technical faithfulness and communicative effectiveness: can the explanation be understood by the intended audience? This motivates work on natural language explanations, counterfactual explanations ("the decision would have been different if X had been different"), and simplified visual summaries.

The safety motivation for interpretability is different in character from debugging and trust. It concerns model failures on current tasks and model behavior in future situations that may differ significantly from training.

As we discussed in the chapters on safety risks, red teaming, and jailbreaking in Part LX, current safety approaches largely work by training models not to produce harmful outputs and by adding defenses at system boundaries. These approaches are empirical: we test the model on many situations and observe whether it behaves safely. But empirical testing cannot cover the full distribution of inputs a deployed model will encounter, and it provides limited guarantees about novel situations. A model that passes all your safety evaluations might still fail on an input type you did not think to test, and in safety-critical contexts that is not an acceptable residual risk.

Interpretability offers a different kind of assurance. If you can examine a model's internal representations and verify that it has not encoded dangerous knowledge or misaligned objectives, you have a more principled basis for safety claims than behavioral testing alone. This vision motivates what is sometimes called "mechanistic safety": understanding the circuits and representations inside models well enough to make verifiable claims about their behavior.

The alignment problem concerns ensuring that a model is pursuing the objectives you intend rather than proxy objectives that correlate with them in training but diverge in deployment. Interpretability is relevant here because a model that has learned misaligned objectives might behave correctly during evaluation while pursuing different goals in deployment. If you can examine the model's internal representations and verify that its intermediate computations reflect the intended objectives, you have stronger evidence of alignment than behavioral testing alone can provide.

Concrete safety applications of interpretability include:

• Identifying whether a model has learned deceptive reasoning patterns, where the stated chain-of-thought diverges from the actual computation.
• Locating knowledge about dangerous capabilities so that knowledge can be removed or suppressed via targeted editing of model parameters.
• Verifying that safety training has modified relevant circuits rather than merely adding superficial filters that can be bypassed with slightly modified inputs.
• Understanding whether a model's stated reasoning in a chain-of-thought reflects its true computational process, or is post-hoc rationalization generated independently of the answer.
• Detecting whether a model has internally represented deceptive strategies, even if it does not currently deploy them.

The gap between current interpretability capabilities and these ambitious safety goals is large. Existing techniques provide partial, often noisy views of model internals. Fully verifying the absence of misaligned objectives in a large language model is well beyond current capabilities. But the safety motivation is a key driver of the research agenda, and progress in interpretability contributes directly to progress in safety.

Beyond practical applications, interpretability is also a scientific goal in its own right. Language models are among the most complex artifacts humans have built. Understanding how they work is intrinsically interesting, in the same way that understanding the brain or the genome is interesting. And scientific understanding often has practical consequences that are not obvious in advance.

Scientific questions about language models include:

• Do language models develop internal representations that correspond to human-legible concepts (syntactic roles, semantic categories, factual entities, logical relations)?
• Does the emergence of capabilities with scale reflect specific structural changes in the network, or is it a smooth statistical effect that only looks discontinuous when measured with the wrong metric?
• How do models perform multi-step reasoning? Do they execute discrete inferential steps, or is the answer retrieved from a distributed representation that encodes the answer implicitly?
• What is the relationship between the training corpus and the model's behavior? Which parts of the training data are responsible for specific capabilities, biases, or failure modes?
• Are the representations learned by different architectures (encoder-only, decoder-only, encoder-decoder) fundamentally different, or do they converge to similar solutions for similar tasks?
• What does it mean for a model to "know" something? Is there a distinction between knowing a fact, being uncertain about it, and being ignorant of it, and can we detect these states internally?

These questions connect to longstanding debates in cognitive science and linguistics about the nature of language understanding and the relationship between statistical learning and structured knowledge. Language models are not just engineering objects; they are interesting models of how statistical regularities in language give rise to semantically rich representations. The findings of interpretability research feed back into theories of language acquisition, compositional semantics, and the cognitive basis of reasoning.

The scientific angle also motivates a kind of rigor that purely practical interpretability work can sometimes lack. Scientific claims need to be reproducible, quantitative, and falsifiable. A claim that "attention head 4 in layer 3 tracks syntactic dependencies" needs to be tested with controlled experiments that distinguish this explanation from alternatives. The scientific framing pushes interpretability research toward stronger experimental standards.

All interpretability goals run into a shared methodological problem: the difficulty of establishing that an explanation is faithful to the model's computational process. This is not a peripheral concern or a technical detail. It is the central challenge that separates useful interpretability work from work that generates plausible-looking but misleading results.

Consider attention visualization, the simplest and most widely used interpretability technique. When a transformer processes a sentence, each attention head computes a weighted average over the input positions. These weights can be visualized as a matrix showing which tokens attend to which other tokens. The visualization is intuitive and easy to generate, and it is tempting to read it as showing which inputs are causally important to the output.

The problem is that attention weights do not straightforwardly indicate which input tokens caused an output. As research discussed in the Attention Analysis Limitations chapter (coming up in this part) has shown, attention weights are not unique determinants of the output. You can modify attention weights significantly without changing the output, and you can achieve the same output through different attention patterns. The output depends on both the attention weights and the values being weighted, and high attention to a token does not mean that token is causally responsible for the output if the value vector at that position carries little relevant information.

An explanation based on attention weights may look informative while misidentifying the actual causal factors. Jain and Wallace (2019) showed that alternative attention distributions that produce the same predictions can be found by optimization, which means many equally valid "explanations" exist. Wiegreffe and Pinter (2019) pushed back, arguing that the question is not whether alternative distributions exist but whether the actual attention distributions are associated with model behavior in a meaningful way. The debate illustrates a deeper tension: explanations can be consistent with the data while being unfaithful to the mechanism.

This faithfulness problem recurs across interpretability methods. Feature attribution methods (like SHAP or integrated gradients) assign credit to input features, but the assignment depends on assumptions about what "credit" means and how to measure it. Different attribution methods produce substantially different attributions on the same input, which raises the question of which one is right. Probing classifiers test whether a layer encodes a concept, but encoding a concept is not the same as using that concept to generate outputs. A model might encode syntactic structure throughout its layers without ever using that structure to determine its outputs, simply because the structure was encoded implicitly during training on a language modeling objective. Mechanistic interpretability traces circuits through the network, but circuits are identified by a combination of evidence that can be ambiguous, and the boundaries between circuits are often unclear.

No method provides a perfect window into a model's internals. A productive way to think about this is that different methods provide different kinds of evidence, each with its own strengths and failure modes. Combining multiple methods and designing experiments that explicitly test faithfulness claims leads to stronger conclusions than any single method alone. The gold standard is causal intervention: if you modify what the explanation says is important, does the output change in the predicted way? Causal experiments are more expensive to run, but they provide substantially stronger evidence than passive observation.

The chapters in Part LIX cover specific interpretability techniques. Here is a map of these methods, organized by what question each technique answers. Understanding this organization before diving into individual techniques helps you see how the field fits together and which methods address which goals.

Attention visualization (Chapter 2) extracts and plots the attention weight matrices from transformer models. These visualizations can reveal broad patterns: some heads specialize in attending to syntactic neighbors, others in long-range coreference, others in positional patterns, and a few in seemingly random patterns that defy easy interpretation. The technique is fast, requires no additional training, and produces visualizations that are intuitive to read.

The subsequent Chapter 3 on Attention Analysis Limitations is equally important: it documents why attention weights are not reliable indicators of causal importance. Understanding both the utility and the limitations of attention visualization is essential for interpreting the literature and applying the method without over-interpreting results. The appropriate use of attention visualization is as a hypothesis generator and a qualitative exploration tool, not as definitive evidence of causal importance.

Probing classifiers (Chapter 4) test whether a particular layer of a model has encoded a particular concept. The method trains a small linear classifier on the activations of a layer to predict some target property, such as part-of-speech tags, syntactic dependencies, or named entity types. If a simple classifier achieves high accuracy, that suggests the concept is linearly encoded in that layer's representation space, meaning the concept is accessible by a simple linear readout from the hidden state.

The simplicity of the classifier is an important design choice. If you use a highly nonlinear classifier, it might succeed by learning the concept itself rather than detecting whether the layer has encoded it. Using a linear classifier constrains the method to detecting only structure that is already explicit in the representation geometry.

Probing layers (Chapter 5) examines how encoding evolves across the depth of the network. For most linguistic properties, shallow layers encode surface features while deeper layers encode semantic and contextual properties. This layered structure is reproducible across model families and provides evidence about how language models build up representations progressively from token identities to discourse-level meaning.

The main limitation of probing is that it tests encoding, not usage. A layer may encode syntactic structure that the model learned incidentally without using it for the task at hand. Control experiments are needed to distinguish encoding from functional use, and researchers have developed several such controls, including comparing probing accuracy to that of a baseline that has access only to label frequency, and testing whether ablating the encoded structure changes model behavior.

Activation patching (Chapter 6) takes a causal approach. Instead of asking "what information is encoded here?" it asks "if I modify this activation, does the output change?" The method runs a model on two inputs, one "clean" and one "corrupted," and patches activations from the clean run into the corrupted run at specific sites. If patching a particular site restores the clean output, that site is causally implicated in the behavior. This causal framing addresses the faithfulness problem more directly than passive observation.

Activation patching has been used to locate where factual associations are stored in transformer networks, identifying specific MLP layers that act as "memory" for particular facts. It has also been used to trace where different linguistic computations happen, with results that broadly confirm the probing finding that syntax and semantics are encoded at different depths.

The logit lens (Chapter 7) is a complementary tool that tracks how predictions evolve across layers. By projecting intermediate activations back into the vocabulary space using the model's unembedding matrix, you can read off what the model is "predicting" at each layer, effectively asking: if the model had to generate a token right now, before seeing the remaining layers, what would it predict? This reveals when the final answer is committed to versus when the model is still processing and when it switches from one answer to another.

Sparse autoencoders (Chapter 8) address a fundamental challenge in neural network interpretation: individual neurons do not cleanly correspond to interpretable concepts. Research has found that neurons often respond to multiple unrelated features simultaneously, a phenomenon called polysemanticity. A single neuron might activate for both the concept "software developer" and the concept "baseball" with no obvious connection between them. This makes interpreting individual neurons unreliable.

Sparse autoencoders are trained on a layer's activations to find a dictionary of features, many more features than neurons, such that each activation can be expressed as a sparse combination of dictionary elements. The resulting features tend to be more interpretable than individual neurons because the sparse constraint forces each feature to be used for fewer distinct concepts. This is a form of dictionary learning applied to neural network activations.

Feature interpretation (Chapter 9) examines what these learned features represent. Automated methods analyze which inputs maximally activate each feature, looking for patterns in the text that explain the feature's behavior. Some features correspond to interpretable concepts (specific named entities, syntactic positions, stylistic properties, emotional tones), while others remain opaque. The proportion of interpretable features varies across layers and model sizes.

Mechanistic interpretability (Chapter 10) attempts to fully reverse-engineer the algorithms that specific components implement. Rather than asking what information is encoded where, it asks: what computation is this circuit performing, expressed as a precise algorithm? The classic example is the induction head circuit: a composition of two attention heads that implements an in-context pattern completion operation, enabling a model to continue a pattern like "A B ... A" with "B" by recognizing the previous occurrence of A and copying the token that followed it.

Mechanistic analysis has produced a small but growing catalog of circuits that implement specific capabilities in transformer models. The approach requires intensive manual work and careful experimental design, but it produces the most detailed understanding of any interpretability method. When mechanistic analysis succeeds, you end up with an algorithm written in terms of matrix operations that you can verify and simulate independently of the neural network.

Activation steering (Chapter 11) applies interpretability findings to influence model behavior. If you can identify a direction in activation space that corresponds to a concept (such as "honesty," "refusal," or "positive sentiment"), you can add a multiple of that direction to activations during generation to steer the model toward or away from that concept. This converts interpretability knowledge into a form of control, closing the loop between understanding and intervention. Activation steering has been used experimentally to suppress harmful outputs, enhance factual accuracy, and modulate emotional tone.

The techniques above serve different goals. A clear taxonomy helps you choose the right tool for the question at hand. Using an attention visualization to verify safety properties, or a probing classifier to explain a specific user-facing decision, is likely to produce misleading results because the method is not suited to the goal.

No single method answers all questions. Real interpretability investigations typically combine multiple techniques, using each to address its most suitable question and cross-validating findings across methods to distinguish robust results from method-specific artifacts.

Interpretability can be applied at different levels of scope, and the appropriate level depends on the question you are trying to answer. Choosing the right granularity is not trivial: going too coarse (model-level analysis) can obscure the mechanisms you want to understand, while going too fine (neuron-level) can produce explanations too detailed to be actionable. Most productive interpretability studies operate at two or three scales simultaneously, using coarse analysis to identify where to look and fine-grained analysis to explain what is found there.

Token level analysis examines the relationship between specific input tokens and output tokens. Feature attribution methods operate at this level, assigning scores to input tokens that indicate their contribution to a particular output. Token-level analysis is the most common form of explanation presented to end users, because it is intuitive: highlighting words in the input that influenced the answer resembles how humans explain their reasoning. The limitation is that token-level attribution does not explain how the model processes those tokens internally. Knowing that the word "Paris" was important to the model's output of "France" tells you that the model is using that information, but not whether it is doing so through a learned association, a factual retrieval, or a chain of intermediate computations.

Layer level analysis examines how representations change as information flows through the depth of the network. Probing classifiers and the logit lens operate at this level, revealing how different properties emerge at different depths. Layer-level analysis has produced some of the most consistent findings in interpretability research: syntactic structure tends to peak in early-to-middle layers, while semantic and world-knowledge properties peak in later layers. These patterns replicate across model families (BERT, GPT, T5), suggesting they reflect something fundamental about how depth interacts with representation quality. The finding is robust enough that it now informs architectural choices: tasks that require syntactic processing can sometimes be handled with smaller models by accessing representations at shallower layers.

Head level analysis examines specific attention heads within a layer. Individual heads tend to specialize in particular patterns (syntactic, positional, semantic, copying), and identifying these specializations is a key step in mechanistic analysis. Early work by Clark et al. (2019) showed that specific heads in BERT reliably tracked syntactic dependencies, direct objects, and coreference relations. Ablating individual heads can test whether they are necessary for a behavior or merely correlated with it, which is an important control for distinguishing causal roles from incidental correlations.

Circuit level analysis examines compositions of multiple components. Induction heads, for example, are a circuit involving two heads across two layers, and understanding the circuit requires understanding how the first head sets up a pattern that the second head completes. Circuit analysis is the most detailed form of interpretability but also the most labor-intensive. It typically combines activation patching, attention head ablations, and manual inspection to reverse-engineer what computation a group of components is collectively performing. The payoff is that circuit-level understanding produces algorithmic explanations, not just correlational ones.

Model level analysis examines global properties of the model, such as overall representation geometry, distribution of factual knowledge across layers, or high-level behavioral properties. Model-level analysis is often the entry point for a new investigation: it tells you where in the model to look before you zoom in. Centered kernel alignment (CKA) similarity matrices computed across all layer pairs, for instance, can reveal whether a model has a distinct "processing phase" in early layers and a "retrieval phase" in later layers before you have examined any individual component. Model-level results are less mechanistically informative than circuit-level results, but they are much faster to obtain and can efficiently narrow the search space.

To make interpretability concrete, let's implement a basic exploration of a small transformer model's internal representations. We will examine how activations change across layers for different types of inputs, a foundation for the probing and visualization techniques in later chapters. This code is exploratory rather than definitive: it demonstrates the setup and measurement tools, but four sentences is far too small a dataset to draw firm conclusions.

We start by loading a pretrained model and setting up extraction of intermediate activations.

These projections illustrate one of the core findings of representation analysis: how sentences cluster in activation space changes across layers. While a six-layer model like DistilBERT-base shows modest differences with only four sentences, larger models like BERT-base and GPT-2 show dramatic stratification, with deeper layers encoding semantic neighborhoods that align closely with human-legible categories. With a larger dataset, you would observe that sentences with similar topics, styles, or syntactic structures group together in final-layer activation space, even when those sentences share no words in common.

A natural question is: how different are representations from one layer to the next? This can be measured with centered kernel alignment (CKA), a metric that compares two representation matrices in a way that is invariant to rotation and isotropic scaling.

The linear CKA between representation matrices $X \in \mathbb{R}^{n \times p}$ and $Y \in \mathbb{R}^{n \times q}$ is:

where:

• $X \in \mathbb{R}^{n \times p}$: the first representation matrix, with $n$ data points and $p$ features, centered by subtracting column means
• $Y \in \mathbb{R}^{n \times q}$: the second representation matrix, with the same $n$ data points and $q$ features, also column-centered
• $\|M\|_F$: the Frobenius norm of matrix $M$, equal to the square root of the sum of squared entries
• $Y^T X$: the cross-covariance matrix capturing how features in $X$ co-vary with features in $Y$
• $X^T X$ and $Y^T Y$: the self-covariance matrices of each representation, serving as normalization terms

The numerator $\|Y^T X\|_F^2$ measures the total squared cross-covariance between the two representations. The denominator normalizes by each representation's own variance, making the score invariant to rotation and rescaling. A score near 1.0 means the two layers encode information in a geometrically similar way, regardless of the specific directions used. This invariance is important because two layers could represent the same information in completely different coordinate directions without CKA detecting a difference, which would happen with naive cosine similarity.

CKA was introduced by Kornblith et al. (2019) to address a fundamental problem in comparing neural representations: two networks or two layers might learn the same information but encode it in different coordinate systems. Direct comparison of weight matrices or activation vectors fails in this case, but CKA succeeds because it measures relational structure (how data points relate to each other) rather than absolute positions.

CKA analysis reveals the transformation profile of the network. Layer pairs with high CKA are doing similar processing across that transition; pairs with low CKA represent significant representational shifts. In larger models, early layers and late layers often show low cross-layer CKA, while middle layers show high similarity, suggesting the bulk of semantic processing happens in a concentrated region of the network rather than uniformly across all layers.

This kind of global analysis is also useful for comparing models. If you compute CKA between corresponding layers of two different models trained on the same task, high values suggest the models have converged to similar internal representations despite starting from different random initializations. This convergence result, observed empirically, suggests that the representations are to some degree determined by the data and the task rather than being arbitrary.

The exploration above uses several parameters worth understanding:

• model_name: The pretrained model to inspect. Swapping in a larger model (e.g., bert-base-uncased) will reveal more pronounced layer differentiation, since larger models build richer intermediate representations. Encoder-only models like BERT and DistilBERT are convenient for this kind of analysis because their representations are bidirectional; with decoder-only models you typically look at the last-token representation rather than CLS.
• output_hidden_states: Must be set to True when loading the model to enable extraction of all layer activations. Without this flag, only the final hidden state is returned.
• n_components (PCA): Controls how many principal components are computed. Two components enable 2D visualization; increasing to 10 or 50 gives a better picture of the full representational geometry, especially when you want to measure variance explained.
• linear_cka: The linear variant of CKA assumes a linear kernel. A kernel CKA variant (using RBF kernels) can capture non-linear similarities but is more computationally expensive and behaves differently on small sample sizes.

The code above is a starting point, not a definitive interpretability analysis. Several important caveats apply.

With only four sentences, the CKA and PCA results are heavily influenced by the specific sentences chosen and should not be generalized. Real interpretability studies typically use hundreds to thousands of examples, and the results are often reported with error bars across multiple random subsets to ensure robustness. The similarity trajectory across layers might look quite different with a different four sentences.

Using the CLS token representation is a reasonable starting point for sentence-level analysis, but it is not the only choice. Other choices include mean-pooling over all token positions, using the last token (as is typical for causal language models), or examining specific tokens that are known to carry particular information. Different choices can reveal different aspects of the representation.

The PCA visualization discards the vast majority of the representation's dimensionality: DistilBERT's hidden size is 768, and the two principal components explain only a fraction of the total variance. Points that look close in the 2D projection might be far apart in the full space, and points that look far apart might be close along dimensions the projection discards. This is a fundamental limitation of all dimensionality reduction methods, and it means you should treat 2D projections as suggestive rather than definitive.

Despite these limitations, the code above illustrates the general approach: load a model, extract activations at each layer, and measure some property of those activations (similarity, clustering, variance explained). This pattern underlies nearly all representation analysis in interpretability research. The specific measurements and scale of the analysis change, but the structure stays the same.

Interpretability research does not exist in isolation from model development. The two activities are in dialogue in ways that are worth understanding.

On one side, model development informs interpretability: as models evolve (larger, more capable, trained on more data), interpretability researchers need to adapt their methods to remain applicable. Techniques that work on BERT may not scale to GPT-4-sized models. Circuit analyses that fully characterize a capability in GPT-2 may find that the same capability is distributed across far more components in a larger model, making full characterization infeasible.

On the other side, interpretability informs model development in increasingly concrete ways. Understanding which layers store factual knowledge (via activation patching) has informed the design of knowledge editing methods that modify those specific layers. Understanding how attention heads specialize has informed efforts to prune models by removing heads that are not useful for a particular downstream task. Understanding polysemanticity has motivated research into architectures that encourage more interpretable, monosemantic representations.

This feedback loop suggests that interpretability can guide model development as well as retrospective analysis. It can inform training choices, architectural choices, and evaluation standards. Models could in principle be designed with interpretability as a constraint, rather than treating it as an afterthought. This direction, sometimes called "inherently interpretable" or "transparency-by-design," is an active research area that trades some raw capability for increased explainability.

To make the goals of interpretability more concrete, consider three common scenarios that arise in practice and how interpretability tools support each one.

Scenario 1: A language model is deployed in a clinical setting and occasionally produces incorrect medication dosage recommendations. The immediate concern is patient safety, and the question is: under what conditions does the model fail, and can the failure mode be characterized precisely enough to add a safeguard? An interpretability workflow might begin with instance-level attribution analysis on known failure cases to identify common patterns (does the model fail when the input uses abbreviations? when the dosage is expressed in unusual units?). It would continue with probing classifiers to check whether the model's layer representations encode quantity magnitudes correctly. If probing reveals that quantity representations are unreliable in specific layers, that is both a diagnostic and a potential target for intervention, either via fine-tuning or by routing quantity-related queries to a specialized module.

Scenario 2: A language model is being evaluated before deployment in a politically sensitive application. The concern is that the model may encode political biases that would systematically disadvantage some groups or perspectives. An interpretability workflow might use probing classifiers to test whether demographic attributes (political affiliation, nationality, religion) are encoded in the model's representations of queries and documents. It would use activation patching to test whether those encodings causally influence downstream outputs. It would use feature analysis on sparse autoencoder features to look for features that combine demographic proxies with evaluative valence. No single method is conclusive, but convergent evidence from multiple methods provides a reasonable audit.

Scenario 3: A researcher wants to understand how a model performs multi-step mathematical reasoning. The scientific question is whether the model executes the steps of a calculation sequentially (each layer implementing one operation) or retrieves the answer directly from a distributed representation of the problem. A workflow might use the logit lens to track when the model commits to an answer across layers: if the final answer appears in early layers for simple problems but only in late layers for complex ones, that suggests sequential processing. Activation patching could test whether intermediate computational states (partial products, intermediate sums) are causally necessary for the final answer. Mechanistic analysis of a small model doing simple arithmetic might fully characterize the circuit, giving a ground-truth algorithm to compare against.

These scenarios share a common structure: start with a hypothesis, choose methods matched to that hypothesis, run controlled experiments to test the hypothesis, and cross-validate across methods. Interpretability is most productive when treated as hypothesis-driven empirical research, not as visualization generation.

Interpretability is a young field with substantial unsolved problems. Being clear about current limitations matters for setting realistic expectations and for identifying where the field most needs to advance.

Scalability. Most detailed interpretability work has been done on small models: GPT-2 (1.5B parameters), BERT-base (110M parameters), and smaller. The computational costs of mechanistic analysis grow with model size, and it is unclear how well techniques that work for small models will scale to frontier models with hundreds of billions of parameters. A circuit that is implemented by five attention heads in a small model might be distributed across hundreds of heads in a larger model, making characterization vastly harder.

Incompleteness. No existing method gives a complete account of a model's computations. Probing tells you about encoding but not usage. Attribution methods identify important inputs but not why they are important. Mechanistic analysis has fully characterized only a handful of circuits in any model, and these circuits account for only a tiny fraction of the model's total computational capacity. The gap between what we can explain and the total behavior of a frontier model is enormous.

Faithfulness verification. Verifying that an explanation is faithful is itself a hard problem. The most rigorous check is causal: if you modify the thing the explanation says is responsible, does the output change as predicted? But this verification is computationally expensive, requires careful experimental design, and may not always be feasible for complex behaviors. For behaviors that emerge from many components jointly, ablating any individual component may not produce the predicted change, even if that component contributes.

Evaluation standards. Unlike supervised learning, interpretability lacks standardized benchmarks. Different papers use different criteria for what counts as a successful explanation, making it hard to compare methods or measure progress objectively. A paper that claims to have "identified the circuit responsible for indirect object identification" may be making a much stronger or weaker claim than it appears, depending on what precisely "circuit" and "responsible" mean in that context.

Polysemanticity. Individual neurons and features in large models tend to respond to many unrelated concepts simultaneously. This makes it harder to assign clean semantic interpretations to model components. Sparse autoencoders (Chapter 8) address this partially, but the underlying polysemanticity reflects fundamental properties of how neural networks use their capacity efficiently. A neural network trained by gradient descent will tend to pack as much information as possible into its available neurons, and superposition of many features onto few neurons is an efficient way to do this.

Reverse engineering vs. prediction. Interpretability aims to understand how a model works, but understanding may not be sufficient to predict or prevent failures. A model that is fully understood in principle might still fail in ways that surprise you in practice, because the space of possible inputs is too large to reason about exhaustively. Understanding the mechanism does not automatically give you the ability to predict all outputs from all inputs.

The gap between description and explanation. Many interpretability findings are more accurately described as descriptions than explanations. Finding that a particular attention head attends to the previous token describes a pattern; explaining why that pattern produces the observed model behaviors requires connecting the description to a causal account of the computations. The transition from description to explanation requires additional experimental work that is often not fully carried out.

Despite these limitations, interpretability research has already produced actionable findings: identification of factual storage sites in MLP layers, characterization of induction head circuits, development of sparse autoencoders that find interpretable features, and causal tracing tools that locate where specific computations occur. The field is making measurable progress, even if the goal of fully understanding large language models remains distant.

Interpretability research aims to understand what is happening inside neural language models, connecting observed outputs to the internal computations that produced them. The goals fall into four main categories:

• Debugging and error analysis uses interpretability to diagnose why models fail on specific inputs and identify systematic errors, enabling targeted fixes rather than blind retraining.
• Trust and accountability uses interpretability to produce explanations for model decisions and audit models before deployment, supporting both user confidence and regulatory compliance.
• Safety and alignment uses interpretability to verify that models have not learned dangerous patterns and to check that safety training is modifying the right internal circuits rather than adding superficial filters.
• Scientific understanding uses interpretability to characterize the representations and algorithms that language models implement, adding to the broader science of how statistical learning produces semantic competence.

The core methodological challenge across all these goals is faithfulness: ensuring that explanations accurately reflect the model's computational process rather than merely being consistent with the observed output. A plausible-sounding unfaithful explanation is worse than no explanation, because it creates false confidence. The gold standard for faithfulness testing is causal intervention: modify what the explanation says is important and verify that the output changes as predicted.

Interpretability methods operate at different granularities, from token-level attribution to circuit-level mechanistic analysis, and the appropriate level depends on the question. Model-level analysis provides an efficient entry point; circuit-level analysis provides the most mechanistically detailed answers. Most real investigations combine multiple levels and multiple methods, using each for its comparative advantage.

The main technical approaches, each covered in depth in the following chapters, are:

• Attention visualization (fast, widely applicable, but limited faithfulness guarantees)
• Probing classifiers (tests whether concepts are encoded; layer probing tracks their evolution across depth)
• Causal intervention methods: activation patching and the logit lens, which test causal roles directly
• Sparse autoencoders (finds interpretable features by decomposing activations into overcomplete dictionaries)
• Mechanistic analysis (reverse-engineers specific circuits to algorithmic precision)
• Activation steering (applies interpretability findings to modify model behavior at generation time)

Each technique has a different strength and is suited to different interpretability goals. As you work through the specific methods in this part, keep in mind what question each technique is designed to answer, what its faithfulness guarantees are, and how it fits into the larger interpretability research program.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about interpretability goals for language models.