Interpretability Goals › The Core Challenge: Identifying Faithful Explanations

All interpretability goals run into a shared methodological problem: the difficulty of establishing that an explanation is faithful to the model's computational process. This is not a peripheral concern or a technical detail. It is the central challenge that separates useful interpretability work from work that generates plausible-looking but misleading results.

Consider attention visualization, the simplest and most widely used interpretability technique. When a transformer processes a sentence, each attention head computes a weighted average over the input positions. These weights can be visualized as a matrix showing which tokens attend to which other tokens. The visualization is intuitive and easy to generate, and it is tempting to read it as showing which inputs are causally important to the output.

The problem is that attention weights do not straightforwardly indicate which input tokens caused an output. As research discussed in the Attention Analysis Limitations chapter (coming up in this part) has shown, attention weights are not unique determinants of the output. You can modify attention weights significantly without changing the output, and you can achieve the same output through different attention patterns. The output depends on both the attention weights and the values being weighted, and high attention to a token does not mean that token is causally responsible for the output if the value vector at that position carries little relevant information.

An explanation based on attention weights may look informative while misidentifying the actual causal factors. Jain and Wallace (2019) showed that alternative attention distributions that produce the same predictions can be found by optimization, which means many equally valid "explanations" exist. Wiegreffe and Pinter (2019) pushed back, arguing that the question is not whether alternative distributions exist but whether the actual attention distributions are associated with model behavior in a meaningful way. The debate illustrates a deeper tension: explanations can be consistent with the data while being unfaithful to the mechanism.

This faithfulness problem recurs across interpretability methods. Feature attribution methods (like SHAP or integrated gradients) assign credit to input features, but the assignment depends on assumptions about what "credit" means and how to measure it. Different attribution methods produce substantially different attributions on the same input, which raises the question of which one is right. Probing classifiers test whether a layer encodes a concept, but encoding a concept is not the same as using that concept to generate outputs. A model might encode syntactic structure throughout its layers without ever using that structure to determine its outputs, simply because the structure was encoded implicitly during training on a language modeling objective. Mechanistic interpretability traces circuits through the network, but circuits are identified by a combination of evidence that can be ambiguous, and the boundaries between circuits are often unclear.

No method provides a perfect window into a model's internals. A productive way to think about this is that different methods provide different kinds of evidence, each with its own strengths and failure modes. Combining multiple methods and designing experiments that explicitly test faithfulness claims leads to stronger conclusions than any single method alone. The gold standard is causal intervention: if you modify what the explanation says is important, does the output change in the predicted way? Causal experiments are more expensive to run, but they provide substantially stronger evidence than passive observation.