Interpretability Goals › What the Code Demonstrates and What It Does Not

The code above is a starting point, not a definitive interpretability analysis. Several important caveats apply.

With only four sentences, the CKA and PCA results are heavily influenced by the specific sentences chosen and should not be generalized. Real interpretability studies typically use hundreds to thousands of examples, and the results are often reported with error bars across multiple random subsets to ensure robustness. The similarity trajectory across layers might look quite different with a different four sentences.

Using the CLS token representation is a reasonable starting point for sentence-level analysis, but it is not the only choice. Other choices include mean-pooling over all token positions, using the last token (as is typical for causal language models), or examining specific tokens that are known to carry particular information. Different choices can reveal different aspects of the representation.

The PCA visualization discards the vast majority of the representation's dimensionality: DistilBERT's hidden size is 768, and the two principal components explain only a fraction of the total variance. Points that look close in the 2D projection might be far apart in the full space, and points that look far apart might be close along dimensions the projection discards. This is a fundamental limitation of all dimensionality reduction methods, and it means you should treat 2D projections as suggestive rather than definitive.

Despite these limitations, the code above illustrates the general approach: load a model, extract activations at each layer, and measure some property of those activations (similarity, clustering, variance explained). This pattern underlies nearly all representation analysis in interpretability research. The specific measurements and scale of the analysis change, but the structure stays the same.