Probing Classifiers › Background: Why Probing?

To understand why probing classifiers became central to NLP interpretability, it helps to understand what alternatives were available before them, and why those alternatives fell short.

The earliest approach to understanding what neural networks learned was weight visualization. In computer vision, you could visualize the convolutional filters of early layers and see that they resembled edge detectors and Gabor filters, which matched well-understood properties of the primate visual cortex. For language models, the equivalent approach would be to look at the weight matrices in word embedding layers. You could find that word vectors clustered semantically in embedding space, that "king - man + woman = queen" held approximately, and that syntactic analogies also worked. But this only worked for the embedding layer. The intermediate layers of a transformer, with their 768-dimensional hidden states produced by interleaved attention and MLP computations, are not interpretable by inspection.

A second approach was ablation: remove parts of the model and see what breaks. If removing an attention head hurts dependency parsing performance on a downstream task, maybe that head is doing dependency-related computation. But ablation is crude. It confounds the information encoded in representations with the model's ability to route and use that information. An attention head might be carrying syntactic information without being the sole processor of it; ablating it might cause minimal performance degradation because other heads compensate.

A third approach was analysis of attention patterns. Much early work visualized which tokens attended to which other tokens and argued that certain heads performed identifiable syntactic operations. Heads that attended strongly from a verb to its subject seemed to be computing subject-verb agreement. This was qualitatively interesting but hard to quantify: how much syntactic information is in the attention pattern? Is the pattern causing the downstream behavior or coincidental?

Probing classifiers offered something different. They provided a quantitative, reproducible method: train a classifier, measure accuracy, report a number. If the number is high, the representations encode the probed property. The method generalizes across any linguistic property for which you have labeled data. It does not require modifying the model, so it can be applied to any pretrained model without retraining.

The first systematic probing studies appeared around 2018 and 2019, coinciding with the rise of BERT. Alain and Bengio (2016) had already proposed the basic methodology for computer vision, and the NLP community quickly adapted it. By 2019, there were dozens of papers probing BERT for POS, syntactic dependency, named entity type, coreference, semantic role labeling, and many other properties. The field had a new vocabulary: probes, probing tasks, probing profiles, and selectivity scores.