Probing Layers › Probing Patterns Across Model Families

Layer probing has been applied to many transformer architectures, revealing that the low-to-high hierarchy of linguistic abstraction is not unique to BERT. However, the precise mapping depends on the model's training objective, architecture, and size.

Encoder models like BERT and RoBERTa are trained with masked language modeling (MLM) on the full sentence. Because every layer can attend to both left and right context, the model can integrate bidirectional signals from the very first layer. This supports early syntactic integration: a token in layer 1 already "knows" what comes both before and after it, which helps establish syntactic category quickly.

Decoder-only models (GPT-style) use causal masking: each token can only attend to previous tokens. This changes the flow of information. The model cannot know what comes after a token until it has processed all preceding context. As a result, probing studies on GPT models often find that syntactic and semantic information develops more gradually and may not be as cleanly localized as in bidirectional models. The profiles tend to be smoother, with less pronounced peaks, because the model has to maintain open uncertainty about future context throughout processing.

An interesting consequence of causal masking is that the representation of a token at a given layer encodes different information depending on what position it occupies in the sequence. A token late in the sequence, after many preceding tokens have been processed, carries richer contextual information than the same token appearing at position 1. This position-dependent variation in representation quality is less pronounced in bidirectional models, which makes GPT-style models somewhat harder to probe cleanly.

Cross-attention in encoder-decoder models (like the original Transformer for translation) adds another dimension. The decoder layers must integrate both their own representations and the cross-attention output over the encoder's top-layer representations. This makes layer probing more complex for these architectures. Probing the encoder layers of a translation model reveals that they develop rich representations of the source language, while probing the decoder layers reveals how the model builds up the target language representation in a way that is conditioned on the encoded source.

Larger models tend to show more gradual, smoother accuracy profiles. In a 24-layer BERT-large versus 12-layer BERT-base, the same linguistic properties still tend to peak in the lower third, middle third, and upper third of the network, respectively. This suggests the functional organization scales with depth rather than being pinned to specific absolute layer numbers.

Larger models also tend to achieve higher peak probe accuracy overall. A 24-layer model's best POS probe outperforms a 12-layer model's best POS probe, partly because larger models have more capacity to encode linguistic information and partly because they are trained on more data. However, the relative improvement from adding depth is often more pronounced for higher-level semantic tasks than for surface syntactic tasks, which saturate quickly even in smaller models.

One particularly interesting scaling result concerns the distribution of linguistic knowledge across layers. In very large models (hundreds of layers or billions of parameters), the functional specialization becomes more pronounced rather than washing out. Individual layers in large models can show extremely sharp probing accuracy peaks, suggesting that certain computations become highly localized as model capacity grows. This is consistent with the view that large models can afford to dedicate specific computational resources to specific subtasks, rather than distributing processing diffusely.

The pre-training objective shapes what information is encoded at which layers. BERT's MLM objective requires the model to predict masked tokens using bidirectional context, which encourages the development of rich syntactic and semantic representations throughout. Models trained with next-sentence prediction or causal LM may develop representations with different distributions across layers.

RoBERTa, trained with more data, longer sequences, and no next-sentence prediction, shows similar but slightly sharper probing profiles than BERT. The representations at each layer tend to be more distinctive, making task localization cleaner. This suggests that the next-sentence prediction objective in original BERT may have introduced some noise that dilutes the quality of the layer-by-layer probing signal.

Instruction-tuned models and RLHF-finetuned variants show an interesting asymmetry: probing accuracy on linguistic tasks may decrease in upper layers relative to the base model. The fine-tuning process re-shapes the upper layers to produce task-appropriate outputs, which can overwrite some of the purely linguistic structure that was present in the pre-trained model. This means that if you want to probe for generic linguistic properties in a fine-tuned model, middle layers are often a safer choice than the final few layers. The middle layers tend to be less affected by instruction fine-tuning because the fine-tuning gradient signal is strongest at the output end of the network.

Contrastively trained models (like SimCSE or sentence-transformers) tell a different story. Because these models are trained to produce high-quality sentence embeddings, the final-layer representations tend to be more semantically structured than in BERT. Probing accuracy for semantic tasks like coreference or semantic textual similarity is often highest at the very top layer in contrastive models, rather than spreading across the upper third as in BERT. This reflects the training objective: a contrastive loss directly encourages the final representation to encode semantic similarity, concentrating semantic information at the output end.