Probing Classifiers › The Interpretation Problem

High probe accuracy is necessary but not sufficient for strong conclusions about what a model encodes. Interpreting probing results correctly requires careful reasoning about three failure modes.

This interpretive challenge is more subtle than it might appear. When you run a probing experiment and find that a linear classifier achieves 90% accuracy on POS tagging using BERT's layer 6 representations, the number raises more questions than it answers. Does it mean BERT "knows" syntax? Does it mean the representations are useful for downstream syntactic tasks? Or does it just mean that logistic regression is good at this, given 768 dimensions of input? Working through these questions carefully is what separates scientifically valid probing conclusions from naive overinterpretation.

The most significant interpretive trap is confusing "the information is accessible in the representation" with "the model uses this information when making predictions." A representation might contain linearly decodable syntactic information that the model's downstream attention and MLP layers never consult. Pimentel et al. (2020) show formally that mutual information between representations and labels is the correct measure of information content, but probe accuracy is only an estimator of this, and an imperfect one.

Consider: if you train a probe on the final layer representations and achieve 90% accuracy on dependency labels, does the model use those dependency labels when predicting the next word? Not necessarily. The syntactic information might be a byproduct of the model learning other statistical patterns. The model might solve its actual task (next-token prediction) through attention patterns that do not respect the syntactic structure you found.

This is why probing results should be interpreted as measuring information content in representations, not as measuring model behavior. The distinction matters for claims about model capabilities. A model that contains syntactic information in its representations is not the same as a model that solves syntactic tasks, and it is not the same as a model that uses syntax to make better predictions.

This gap between information presence and information use has motivated the development of causal interpretability methods, which you will encounter in the chapter on activation patching. Causal methods ask not "is the information there?" but "does the information causally affect predictions?" These are different questions that require different experiments.

A subtler issue is that different probing classifiers have different capacities to extract information, independent of how much information is there. A larger MLP with more parameters might extract more because it is doing more computation itself. A linear probe with regularization might fail to extract information that is clearly present because the regularization is too aggressive.

Voita and Titov (2020) formalize this as the minimum description length (MDL) approach to probing. Instead of measuring accuracy, they measure how many bits are needed to transmit the probe's predictions given the representations. A good probe needs few bits because the representations already encode the task. A poor probe, or one learning from scratch, needs many bits. This gives a more principled measure of how easily extractable the information is, controlling for the probe's own computational power.

Consider what happens as you increase the regularization strength of a logistic regression probe (decrease the parameter $C$). With weak regularization, the probe can fit almost any patterns in 768-dimensional space, including patterns that arise from random noise. With strong regularization, the probe is forced to use only the most statistically robust directions in the representation space. The "true" amount of information is fixed (it is a property of the representations), but the measured accuracy varies with the regularization setting. This is not a flaw in the experiment; it is a feature. Strong regularization tells you about the most robustly encoded information, which may be more meaningful than the total extractable information.

A third problem is that the training data for the probe may be statistically dependent on factors that correlate with the labels, independent of what the model learned. If the probe training set contains many sentences where nouns are always followed by verbs, the probe might learn to exploit the position of the extracted token as a proxy for the POS label. This is especially problematic when the text used for probing comes from the same corpus as the model's pretraining data.

Good experimental design mitigates this by shuffling, stratifying, and using train/test splits that control for superficial correlates. Cross-domain evaluation is particularly important: train the probe on one domain (news text) and test on another (Wikipedia), making sure that corpus-specific statistics don't substitute for linguistic structure.