Probing Classifiers › Limitations and Impact

Probing classifiers have transformed how researchers think about what language models learn, but the methodology has real limits that practitioners should understand.

The most important limitation is the gap between information accessibility and information use. Even if you prove that syntactic tree structure is linearly encoded in BERT's middle layers, this does not tell you whether BERT's attention heads or MLP layers consult this structure when making predictions. It is possible that the syntactic information is a byproduct of the model solving its pretraining task and is never downstream-relevant.

This matters for practical interpretability. If you are trying to understand why a model made a specific prediction, finding that some representation encodes some property is not enough. You need to show that the property causally influenced the prediction. Methods like activation patching (which we explore in the Activation Patching chapter) and causal mediation analysis address this gap directly. Probing tells you about the information in the representation; causal methods tell you about the information used in the computation.

The distinction also matters for model comparison. If Model A encodes syntactic structure and Model B does not, but both perform identically on downstream tasks, what can you conclude? You cannot conclude that syntactic structure is irrelevant, because Model B might have found a different route to the same performance. But you also cannot conclude that Model A is "better" at syntax in any practical sense, because the syntax it encodes may not be consulted.

Another limitation is the probe-complexity confound: more powerful probes can extract more information, but it becomes unclear whether the information was in the representations or whether the probe learned it. Choosing a linear probe is a partial solution, but the choice of regularization strength also affects what the probe can memorize. Weak regularization allows memorization; strong regularization prevents extraction of complex structure.

There is no universally agreed-upon probe architecture or regularization strategy. Papers use different settings, which makes comparing results across papers hazardous. The MDL probing framework is a promising step toward a principled measure, but it has not yet become the default.

Probing experiments require labeled datasets, and linguistic annotations are expensive to produce. Many probing papers use relatively small datasets (thousands of examples) compared to the training data of the models being probed (billions of tokens). This creates a mismatch: the model may have learned subtle linguistic structure from massive data that small probing datasets cannot detect.

Label imbalance is also a problem. If 80% of tokens are nouns and verbs, a probe that learns to output only those labels achieves high accuracy without capturing the full POS system. Stratified sampling and macro-averaged F1 scores help but do not fully solve the problem. Researchers should report both overall accuracy and per-class accuracy or macro-F1 to ensure that rare categories are evaluated fairly.

Dataset construction choices also matter. If all sentences in the probing dataset come from the same source as the model's pretraining corpus (news articles, Wikipedia), the probe may exploit corpus-specific statistical patterns rather than test linguistic information structure. Ideally, probing datasets should be drawn from a different distribution than the pretraining data to test whether the representations generalize.

A more subtle limitation is that most probing research has focused on English. When probing is extended to morphologically rich languages (Finnish, Turkish, Arabic), the picture changes. These languages encode far more information in the form of words, including case, number, gender, tense, and aspect, and the question of whether models trained primarily on English data encode this information is both practically important and theoretically interesting.

Cross-lingual models like mBERT (multilingual BERT) have been probed for their encoding of linguistic properties across languages. The findings show that mBERT encodes some universal linguistic properties consistently across languages, while language-specific properties are encoded in a more language-specific subspace of the representation. This has implications for transfer learning: properties encoded in language-universal subspaces are the most transferable to low-resource languages.

Despite these limitations, probing classifiers have had substantial impact. They gave researchers their first systematic tool for asking "what did this model learn?" without modifying the model or its training procedure. The finding that BERT's representations contain POS, syntactic, and semantic information was not obvious and changed how practitioners thought about pretraining. It explained why fine-tuning on a small labeled dataset could achieve strong performance: the model had already learned much of the linguistic structure it needed, and fine-tuning only needed to align that structure to the target task.

Probing also clarified the design space for model improvements. If probing shows that a model fails to encode some linguistic property, you can diagnose whether to add pretraining objectives, modify tokenization, or change the model architecture. The finding that standard BERT fails to encode certain types of temporal reasoning motivated pretraining objectives that expose the model to more temporal structure in the text.

The broader intellectual contribution is a conceptual vocabulary for discussing what language models learn. Before probing, the dominant framework was input-output behavior: does the model perform well on task X? After probing, researchers could ask more specific questions: which layer encodes property Y, how robustly, with what selectivity, and in what geometric structure? This vocabulary has shaped both academic research and practical interpretability work in industry.