Probing Layers › Limitations of Layer Probing

Layer probing is powerful, but interpreting its results requires care. Several important limitations have been identified, and understanding them is as important as understanding the methodology itself.

Finding that a probe can decode property $P$ from layer $l$ does not mean layer $l$ uses property $P$ to make decisions. There is an important distinction between information being present (encoded in a representation) and being causally relevant (influencing outputs). A property could be encoded in the representation as a kind of side effect of other computations, without being directly used downstream.

Consider the analogy of a photograph of a person. The photograph encodes a great deal of information about the person (height, hair color, clothing, approximate age) that is not relevant to any particular task you might use the photograph for. A classifier trained to detect one feature (say, gender) from the photograph would succeed, but that does not mean the feature caused the photograph to exist or that it influences what any other classifier does with the photograph. Similarly, a linguistic property encoded in a BERT representation may be there incidentally, not because the model relies on it for its masked language modeling task.

This is why probing is often paired with other interpretability methods. Attention analysis can show where the model is looking. Gradient-based attribution can show which input tokens most influenced a decision. Activation Patching (which we will explore in the next chapter) can directly test whether manipulating a layer's representation changes model behavior. Probing alone provides a correlational, not causal, picture.

The causal gap is particularly important when drawing conclusions about model behavior. If you find that layer 8 encodes coreference information, you might conclude that the model "uses" coreference to resolve pronouns. But the model might be making those pronoun resolution decisions based on attention patterns that implement a completely different algorithm, and the coreference signal at layer 8 might just be a byproduct of that computation. Causal interventions (patching, ablations) are required to close this gap.

A probe with too much capacity can succeed by doing its own feature learning rather than reading off linearly encoded information. A very deep neural probe trained on BERT representations might achieve high accuracy on every task at every layer, not because BERT encodes everything everywhere, but because the probe can extract and combine information that is only implicitly available.

The solution is to use the simplest possible probe (linear, or at most a single hidden layer with strong regularization) and to compare against a control task with randomly shuffled labels. If accuracy on the shuffled-label task is significantly above chance, the probe has too much capacity and is memorizing rather than decoding.

The shuffled-label control is a critically important diagnostic. When you shuffle the labels in your probing dataset and train the same probe, you are creating a task that requires memorization rather than generalization. A well-behaved linear probe should achieve close to chance accuracy on shuffled labels (since there is no linear structure to exploit). If it achieves substantially higher accuracy, the probe is memorizing the training set, which means the accuracy numbers on the real task are also inflated by memorization.

A complementary control is the random model baseline: probe a randomly initialized (untrained) BERT model. The random model's representations have no linguistic content, so any probing accuracy above chance must come from the probe's ability to exploit statistical regularities in the dataset (like class imbalance) rather than from the representations. Comparing the trained BERT probing results to the random baseline gives a cleaner estimate of how much information is specifically contributed by pre-training.

Probe accuracy depends on the distribution and size of the training data. Small probing datasets may not have enough examples of rare categories to reliably measure accuracy. Imbalanced class distributions can cause the probe to default to the majority class. Any interpretation of layer probing results should account for the composition and size of the probe dataset.

One often-overlooked issue is the relationship between the probe dataset and the pre-training data. If the probing sentences come from the same domain as the pre-training corpus (e.g., news text), the model's representations may be especially well-tuned for that domain. Probing results from out-of-domain sentences (medical text, legal documents, social media) might tell a different story about which layers encode which properties, because the model has seen less training data from those domains.

Dataset construction choices can also introduce spurious correlations. If all the LOCATION entities in your NER probing dataset happen to appear at the end of sentences, and if position is weakly predictive of entity type, then a probe for LOCATION might succeed partly by detecting sentence position rather than entity-type information. Careful dataset construction, with cross-validation across different sentence positions and syntactic structures, is necessary to avoid these artifacts.

A critical subtlety is that BERT uses residual connections. The hidden state at layer $l+1$ is not computed from scratch; it is the sum of layer $l$'s output and the attention/FFN sub-network's output. This means information from early layers is directly propagated to all later layers. When a probe at layer 7 correctly identifies a POS tag, it might be reading off information that was computed at layer 2 and simply passed forward through residual connections.

Some researchers have proposed removing the residual by probing only the delta added by each layer (the FFN or attention output before it is added back). This gives a cleaner picture of what each layer contributes rather than what all preceding layers have accumulated.

The residual dependency problem makes it difficult to attribute specific computations to specific layers. If you observe that probe accuracy for POS tagging is high from layer 2 onward and stays high through layer 12, you cannot conclude that all those layers are computing POS-relevant information. It is equally consistent with the hypothesis that layer 2 computed POS information once, and layers 3-12 simply carried it forward in the residual stream without further modifying it. Probing the deltas rather than the full activations provides a way to test this hypothesis directly.

Another approach to this problem is ablation probing: after identifying a layer that encodes a property, remove that layer (by replacing its output with a zero tensor or a bypass) and measure whether downstream probe accuracy drops. If removing layer 2 causes POS probe accuracy to drop substantially at all subsequent layers, that confirms layer 2 as the primary site of POS computation rather than merely a conduit.

Beyond the methodological limitations discussed above, there is a more fundamental question about what probing tells us. Probing measures linear accessibility of information: whether a hyperplane in representation space separates the classes. But many computationally important forms of information might be encoded non-linearly, or distributed across multiple representations in a way that no single token's representation captures.

For instance, syntactic agreement between a subject and a verb requires relating two token representations. A probe at a single token cannot capture this relational information directly (though it might capture it indirectly if the subject's representation has been updated to reflect agreement requirements). Probing for relational properties typically requires probing over pairs of representations, which increases both the search space and the risk of the probe capacity problem.

Probing also says nothing about the mechanism by which the model uses the encoded information. Even if layer 8 perfectly encodes coreference clusters (high probe accuracy), you do not know whether the model computes coreference by attending to relevant tokens in earlier layers, by matching entity representations stored in FFN weights, or through some combination of both. Mechanistic interpretability tools, which we cover in subsequent chapters, are needed to answer mechanism questions that probing cannot address.