Probing Layers in Transformer Models

When a transformer processes the sentence "The bank by the river flooded," how does it figure out that "bank" means a riverbank and not a financial institution? The model must resolve this ambiguity somewhere inside its stack of layers, and identifying where that resolution happens is exactly what layer probing sets out to answer.

As we saw in the previous chapter on Probing Classifiers, a probe is a simple supervised model trained on top of frozen representations. The key insight is that if a linear probe can predict a linguistic property from a set of representations, those representations encode that property. Layer probing extends this idea across all layers simultaneously: instead of probing a single fixed layer, you extract representations from every layer in the model and train a separate probe at each one. The resulting accuracy profile across layers tells you which layers are responsible for which types of linguistic or semantic information.

Think of it like excavating an archaeological site by taking soil samples at different depths. The artifacts you find at each stratum tell you about the civilization that existed at that level. Layer probing takes the same approach with a neural network: you sample the model at each computational depth and examine what kinds of linguistic knowledge are deposited there. Just as earlier strata hold older, more primitive materials while deeper strata hold refined artifacts from more complex societies, earlier transformer layers tend to hold coarser linguistic features while later layers accumulate richer semantic meaning.

This technique has become one of the most widely used tools in interpretability research. It lets you ask targeted questions like: "At which layer does BERT start distinguishing parts of speech?" or "Does syntactic structure emerge before semantic meaning, or do they appear together?" The answers have shaped how researchers think about the internal organization of transformer models and have practical implications for tasks like transfer learning, model compression, and parameter-efficient fine-tuning. Understanding where information lives inside a model is the first step toward understanding how the model makes decisions, and ultimately toward building more transparent, reliable systems.

The central appeal of layer probing lies in its simplicity: you do not need to modify the model, understand its internal computations, or make assumptions about how it works. You just ask a question about each layer's output and observe whether the answer is accessible. This non-invasive quality makes probing studies widely reproducible and applicable to any architecture that produces layer-by-layer hidden states.

Before diving into probing methodology, it helps to understand what each layer in a transformer is doing. A transformer encoder like BERT consists of a stack of $L$ identical blocks. Given an input sequence of $n$ tokens, each block turns a matrix of hidden states $\mathbf{H}^{(l)} \in \mathbb{R}^{n \times d}$ into a new matrix $\mathbf{H}^{(l+1)}$ of the same shape.

Each layer applies two sub-operations:

• Multi-head self-attention: Each token attends to all other tokens and gathers contextually weighted information.
• Position-wise feed-forward network (FFN): An MLP applied independently to each token's representation, often interpreted as a key-value memory store.

The transformation is:

where:

• $\mathbf{H}^{(l)}$: the hidden state matrix at layer $l$, shape $n \times d$
• $\text{Attn}$: the multi-head self-attention operation
• $\text{FFN}$: the feed-forward sub-network
• $\text{LayerNorm}$: layer normalization applied before each sub-operation (in the pre-norm variant)

Layer 0 is a special case: it is the embedding layer, which maps each token to a $d$-dimensional vector using a learned lookup table plus a positional encoding. This layer has no attention mechanism and no contextual mixing; each token's representation depends solely on its identity (and position), not its neighbors. Every subsequent layer introduces context by allowing tokens to communicate through attention.

The architecture's design has a direct consequence for probing: because each layer builds on the output of the previous one, representations accumulate information progressively. The embedding layer knows only the token identity. Layer 1 can see immediate neighbors through attention. By layer 6, a token's representation has been shaped by six rounds of contextual integration, with the attention patterns in each round potentially attending to very different parts of the sequence.

The residual connections deserve special mention here. In the formula above, each sub-operation adds its output back to the input: $\mathbf{H}^{(l)} + \text{Attn}(\mathbf{H}^{(l)})$. This means information from every preceding layer is directly available at every subsequent layer without needing to be re-encoded. A feature computed at layer 2 can flow through the residual stream untouched all the way to layer 12 if no subsequent layer overwrites it. This has important consequences for interpreting probing results, which we will return to in the limitations section.

The FFN sub-network, while often described as a simple two-layer MLP, has been given a more interesting interpretation in recent years. Geva et al. (2021) showed that FFN layers act as key-value memories: the first weight matrix stores patterns (keys) while the second stores factual content (values). When a key pattern matches the current token representation, the corresponding value is retrieved and added to the representation. This means upper-layer FFNs may be storing and retrieving factual knowledge (like "Paris is the capital of France"), while lower-layer FFNs may be storing and retrieving linguistic patterns (like "a token that looks like a past-tense verb ending in -ed"). Layer probing can detect both types of stored information, though it cannot easily distinguish memory retrieval from attention-based contextual integration.

This progression from context-free to context-rich representations is what layer probing exploits. By training probes at each layer, you can pinpoint the transition from "this token is the word bank" to "this token is the word bank used as a geographical feature."

A major finding from layer probing research (Peters et al., 2018; Tenney et al., 2019; Jawahar et al., 2019) is that linguistic structure is not spread uniformly across a transformer's layers. Instead, different types of information tend to concentrate at different depths.

The general pattern, observed across BERT and similar models, is:

• Lower layers (1-4): Local, surface-level features dominate. Part-of-speech tags, morphological properties (plural vs. singular, verb tense), and basic syntactic categories are most accurately predicted by probes at early layers.
• Middle layers (5-8): Syntactic structure deepens. Dependency parsing accuracy peaks here, suggesting that the model builds up phrase structure and grammatical relations in mid-network layers.
• Upper layers (9-12): Semantic and task-specific information concentrates. Coreference resolution, named entity recognition, and sense disambiguation tend to be best predicted from representations in the final third of the network.

This hierarchy mirrors the pipeline of classical NLP systems: tokenization and tagging come first, parsing comes next, and semantic interpretation comes last. The fact that a model trained purely on masked language modeling recovers this hierarchy suggests the ordering is not arbitrary. Rather, it reflects a natural dependency: to determine the meaning of a phrase, you first need to know its syntactic role.

The intuition is straightforward once you think about what the masked language modeling objective demands. To predict a masked word, the model needs information at multiple levels of abstraction. Knowing that the blank is a noun helps, but knowing that it is the subject of a transitive verb narrows things further, and knowing that the surrounding context involves rivers and flooding narrows things even more. The training signal rewards the model for building up exactly this hierarchy of signals, layer by layer, because each additional level of structure provides better constraints on what the missing word could be.

Syntactic information, especially constituency structure, emerges in middle layers. The concept of syntactic tree depth, which measures how deeply nested a token is in a parse tree, peaks in predictability around layers 5-7 in a 12-layer BERT model. This means the representations at those layers carry implicit structural information about the grammatical organization of the sentence.

Subject-verb number agreement is a classic probe target. Given a sentence like "The keys to the cabinet are on the table," the model must track that the verb "are" agrees with "keys" (the subject) rather than "cabinet" (the nearer noun). Agreement probing accuracy tends to peak in middle layers, suggesting these layers actively maintain grammatical bookkeeping.

What makes syntactic probing informative is that BERT was never explicitly trained on parsing tasks. It was trained only to fill in masked words. The fact that it learns to encode syntactic structure as a byproduct of this objective suggests that syntactic structure is useful for predicting words in context. A model that does not know that "keys" and "are" must agree would struggle to predict the correct verb form in ambiguous sentences. The model thus acquires grammatical knowledge as an instrumental goal rather than an end in itself.

Dependency relations add another dimension. Probing studies can test whether a given token's representation "knows" which other token it syntactically depends on. This is typically framed as a probing task over token pairs: given the representations of two tokens, can a probe determine whether one is the syntactic head of the other? Accuracy on this task also peaks in middle layers and tends to be higher for representations that attend to structurally related tokens.

Constituency structure (the nested phrase groupings) follows a slightly different trajectory than dependency structure (the head-dependent arcs between individual tokens). Constituency tends to emerge earlier, with phrase-level groupings becoming accessible by layer 4-5, while the more complex relational structure captured by dependency graphs requires a few more layers of computation. This makes sense structurally: recognizing that "quick brown fox" is a noun phrase is a relatively local operation, while determining the grammatical role of "fox" with respect to "jumps" requires integrating information over the full clause.

Semantic content, including word sense disambiguation, semantic role labeling, and coreference, concentrates in upper layers. Coreference resolution requires understanding which noun phrases refer to the same real-world entity across a discourse. This is a deeply semantic task that requires integrating information across sentences, and probes at upper layers systematically outperform those at lower ones.

Named entity recognition (NER) shows a different pattern. Surface string matching ("This token looks like a name") may be captured at lower layers, while the actual entity type (PERSON vs. ORGANIZATION vs. LOCATION) becomes clearer in upper layers as the model integrates contextual evidence. The word "Washington" could refer to a person, a state, a city, or a university. Resolving which sense is intended requires reading the surrounding context, which is why entity typing signals are strongest in the upper layers where the most contextual integration has occurred.

Word sense disambiguation provides one of the cleanest illustrations of the upper-layer semantic hypothesis. In the bankrupt/riverbank example we started with, the word "bank" has the same form in both uses but very different meanings. Early layers produce nearly identical representations for both uses, because the token itself is identical and local context may not yet be sufficient to disambiguate. As the model processes deeper layers, the representations diverge: the riverbank "bank" starts to look more like "shore," "river," and "flood" in embedding space, while the financial "bank" starts to resemble "loan," "account," and "interest." This divergence of polysemous words across layers is one of the most direct pieces of evidence that upper layers perform semantic disambiguation.

Semantic role labeling (SRL) asks whether a given word is the agent, patient, instrument, or beneficiary of an event. The question "Who did what to whom?" requires understanding both grammatical structure and semantic content. SRL probing accuracy follows a gradient that peaks even later than syntactic probing, often in the final two or three layers of a 12-layer model. This positions SRL at the top of the linguistic hierarchy as one of the most demanding properties for a probe to decode from linear features of the representation.

Below syntactic properties lies a layer of surface and morphological features that are even more strongly localized to early layers. Morphological probing tests whether a token's representation encodes its morphological properties: whether it is plural, past tense, comparative, possessive, and so on. These properties are, in some sense, encoded directly in the token's surface form ("cats" is clearly plural; "walked" is clearly past tense), so it is not surprising that probes at even layer 1 can often predict them with high accuracy.

What is more interesting is the relative accuracy profile. Layer 0 (the embedding layer) already captures substantial morphological information because the token embeddings for "cat" and "cats" are nearby in embedding space, having co-occurred in similar contexts during training. But there is a measurable jump in accuracy from layer 0 to layer 1, suggesting that even one round of attention improves the representation's morphological signal, possibly by gathering evidence from surrounding words that constrain the likely morphological form.

Sentence length, token position, and other purely structural properties also tend to be most accessible from early layers. These are the properties that require no understanding of meaning, only of form, and the model encodes them early because they are easy to read off from the surface.

One of the most conceptually important aspects of layer probing is what it reveals about how representations evolve through the network. A token's representation at layer 0 is purely lexical: it encodes the identity of the word. As information flows through successive layers, the representation undergoes a continuous transformation, accumulating contextual signals.

A useful mental model is to think of the representation $\mathbf{h}_i^{(l)}$ of token $i$ at layer $l$ as carrying increasingly rich information:

• Layer 0: "I am the token bank"
• Layer 3: "I am bank, a noun, and probably the subject of something"
• Layer 7: "I am bank, a noun, the subject of flooded, which suggests a geographical object"
• Layer 11: "I am bank, meaning a riverbank, playing the LOCATION semantic role in this event"

This evolution is not discrete or stepwise. Probing accuracy profiles are usually smooth functions of layer depth, suggesting gradual refinement rather than abrupt phase transitions. However, there are often local peaks and plateaus that indicate where specific types of computation are concentrated.

The continuous nature of this evolution has an important implication: no single layer "knows" the full answer. The model does not flip a switch at layer 8 and suddenly understand the meaning of the sentence. Instead, each layer adds a small increment of contextual understanding, like a growing blur of context sharpening into focus. This incremental enrichment is consistent with the residual architecture, where each layer adds a small correction to the incoming representation rather than recomputing it from scratch.

Researchers have also observed that representations from neighboring layers tend to be highly similar (cosine similarity often above 0.95) while representations from distant layers can diverge substantially. This suggests that each individual transformer block contributes a modest refinement, but the cumulative effect of many such refinements is dramatic. The embedding layer representation and the final layer representation of the same token often point in very different directions in the 768-dimensional space, even though consecutive layers look almost identical.

One intriguing finding concerns the way contextual information "flows" into a representation. Early layers, even with bidirectional attention, tend to integrate information primarily from nearby tokens. This is partly because attention weights in early layers are often more diffuse and locally biased. Upper layers can exhibit long-range attention patterns that integrate information from tokens many positions away. This means that the disambiguation of "bank" in a long sentence might require upper layers that can attend to a word ten positions back describing the setting as a river valley.

A complementary way to study representation evolution is to measure how much the representation of a token changes from one layer to the next. One approach uses cosine similarity:

where:

• $\mathbf{h}_i^{(l)}$: the representation of token $i$ at layer $l$
• $n$: the number of tokens in the sequence
• $\text{sim}(l)$: the average cosine similarity between consecutive layers $l$ and $l-1$

Low similarity means the representation changed a lot between those two layers. This can identify "active" layers where significant computation happens, as opposed to "pass-through" layers that barely modify their input. In BERT-base, researchers have found that the middle layers tend to show lower inter-layer similarity, suggesting they do more representational work than early or late layers.

Another measure is angular distance, the angle between consecutive layer representations:

where $\theta(l) \in [0, \pi]$. A value near 0 means the representation barely changed; a value near $\pi/2$ means a 90-degree rotation, indicating substantial transformation.

The inter-layer similarity analysis complements probing by providing a different diagnostic lens. Probing tells you what is accessible from a given layer. Inter-layer similarity tells you where the most active computation is happening. Ideally, you would observe that layers with the most change in similarity also correspond to layers where new linguistic properties become accessible in probing. This correspondence is not always perfect, partly because the residual stream means changes in the delta do not necessarily translate to changes in the full representation, but it provides a useful cross-check on probing interpretations.

The residual connection architecture means that we can think of the model as maintaining a shared "residual stream" that all layers read from and write to. Each layer reads the current state of the stream, computes a small additive update (the output of the attention and FFN sub-operations), and writes that update back to the stream. The final representation at layer $L$ is the sum of all the delta updates from layers 0 through $L$.

This perspective has a direct implication for probing. When a probe at layer 7 correctly classifies a token's POS tag, it might be reading information that was computed at layer 2 and simply accumulated unchanged in the residual stream ever since. The probe cannot distinguish between information actively computed at layer 7 and information passively carried from layer 2. This is why some researchers have proposed probing the residuals rather than the full layer outputs: instead of probing $\mathbf{H}^{(l)}$, probe only the delta $\Delta^{(l)} = \mathbf{H}^{(l)} - \mathbf{H}^{(l-1)}$. The delta probe isolates what each layer contributed, rather than what all preceding layers accumulated.

The basic layer probing setup, where you train a linear probe at each layer and plot accuracy, has spawned a rich family of methodological extensions that address its limitations or ask more refined questions.

John Hewitt and Christopher Manning introduced structural probing (2019) as a more geometrically sophisticated alternative to linear classification probes. Instead of asking whether a linear classifier can predict discrete labels (like POS tags), structural probing asks whether a linear transformation of the representation space can recover the entire syntactic tree structure.

The structural probe learns a matrix $\mathbf{B}$ such that the squared distance between transformed representations of two tokens $i$ and $j$ approximates the tree distance between them in the ground-truth parse:

where:

• $d_T(i, j)$: the number of edges in the shortest path between tokens $i$ and $j$ in the dependency parse tree
• $\mathbf{B}$: a learned linear transformation (projection matrix)
• $\mathbf{h}_i, \mathbf{h}_j$: the representations of tokens $i$ and $j$

This is a more demanding test than classification probing. It asks not just "can a probe distinguish noun from verb?" but "does the geometry of the representation space organize tokens in a way that mirrors the topology of their syntactic relationships?" If the answer is yes, then syntactic structure is not just implicitly encoded; it is geometrically structured in the representation.

Hewitt and Manning found that BERT's representations at middle layers pass this structural test with surprisingly high fidelity. The representation space at layers 5-7 can be linearly transformed to approximately recover parse tree distances, suggesting that BERT's representations have internalized tree structure as a geometric property. This is a much stronger claim than what linear classification probing can make: syntactic relationships between tokens are encoded as distances in the representation space, which is a geometric analog of the symbolic parse tree.

The structural probe also introduced a different notion of what "probing" means. Rather than training a classifier on fixed representations, you are fitting a geometric transformation to the representation space. The probe's success tells you about the intrinsic geometry of the representations, not just their predictive content.

A significant challenge with standard probing is deciding when a probe is "too complex." A probe with more parameters can always achieve higher accuracy by memorizing, which inflates the measure of how much information is encoded. Pimentel et al. (2020) proposed an information-theoretic framework that addresses this directly.

In this framework, you measure the mutual information $I(P; R)$ between the probe target $P$ (e.g., POS labels) and the representation $R$ (e.g., the hidden states). Mutual information captures how much knowing the representation reduces uncertainty about the target, regardless of how that information is geometrically organized. The advantage is that mutual information is probe-agnostic: it measures the information content of the representation, not the capacity of the probe to access it.

In practice, estimating mutual information from finite samples requires approximations, often done by using the negative cross-entropy of a trained classifier as a lower bound on the mutual information. This brings the approach back to training classifiers, but with a different interpretive frame: the classifier's performance is now a lower bound on the mutual information rather than a direct measure of linear accessibility.

The information-theoretic view clarifies the relationship between probe complexity and interpretation. A linear probe with high accuracy on a non-linearly encoded property would have a high negative cross-entropy (suggesting high mutual information) even though the property is not linearly accessible. A probe that fails because the property is absent would have low mutual information regardless of probe complexity. This framework provides cleaner separation between "the property is not encoded here" and "the property is encoded non-linearly."

Voita and Titov (2020) introduced minimum description length (MDL) probing as another way to address the probe complexity problem. The key insight is that a good representation should allow a probe to classify correctly with a concise model. If you need to transmit a probe model along with the representation to achieve correct predictions, the cost of that transmission (the description length) is a measure of how non-obvious the encoding is.

MDL probing frames the task as a compression game. You want to find the probe that, together with the representation, minimizes the total description length of the probe and the labeled data. A representation that encodes a property linearly requires a very simple probe (short description length). A representation where the property is deeply buried requires a complex probe (long description length). The MDL framework automatically penalizes complex probes, which prevents inflation from memorization.

In practice, MDL probing is implemented by measuring how many bits are needed to transmit the probe (using online coding schemes) and comparing this to the bits needed to describe the data without the probe (using a uniform prior). The ratio gives a measure of how much the representation compresses the task. This technique has produced more reliable layer localization results than accuracy-based probing on small datasets, where overfitting is a concern.

Running a layer probing study requires answering several design questions. Choices made here significantly affect the conclusions you can draw.

The most common choice is a linear probe: a single-layer classifier with no hidden layers. A linear probe tests whether the target property is linearly decodable from the representation, which is a strict test. If a non-linear probe does much better than a linear one, the property is encoded in a more complex geometry.

You should generally prefer linear probes for interpretability purposes. A complex probe might succeed by doing its own feature engineering, so a high accuracy number no longer tells you much about the representation; it tells you more about the probe's capacity. The canonical recommendation from Alain and Bengio (2017) is to keep probes simple enough that their success is attributable to the representation, not the classifier.

There is an important nuance here. A linear probe is not just simple; it makes a specific geometric claim. Linearly decodable means the property can be read off by a hyperplane in the representation space. This is the mildest possible version of "the information is there": if the property is present at all in any accessible form, a linear probe should be able to find it. If even a linear probe cannot, the property is either not encoded or is encoded in a fundamentally non-linear way that resists linear readout.

Single-hidden-layer MLPs with a small number of units (say, 50-200) are sometimes used when you want to test for slightly non-linear encodings without giving the probe too much freedom. The key constraint is regularization: strong L2 regularization or dropout prevents the MLP from memorizing the training set, which keeps the accuracy signal meaningful.

When probing at multiple layers with the same probe architecture, you should use the same hyperparameters and regularization at every layer. Varying regularization across layers would confound probe capacity differences with differences in how information is encoded.

A good probe task has the following properties:

• Well-defined labels: Each token or sentence should have a clear, unambiguous gold label.
• Linguistically interpretable: The task should correspond to a concept you care about (POS tag, NER label, syntactic role).
• Appropriate difficulty: Very easy tasks may show 99% accuracy at every layer, telling you nothing about localization.

Common probe tasks include:

One underappreciated issue is the relationship between class frequency and probe accuracy. POS tagging in English is dominated by a few frequent tags (NOUN, VERB, DET), and a probe that simply predicts the most common tag will achieve non-trivial accuracy by chance. Reporting only raw accuracy can therefore be misleading. Macro-averaged F1 across all classes, or accuracy stratified by tag frequency, gives a more reliable picture of what the probe has learned.

Another consideration is probe dataset size. Small datasets may not have enough examples of rare categories to produce reliable accuracy estimates. NER probing with 500 examples, for instance, might see very few ORGANIZATION entities and produce noisy accuracy for that class. Larger evaluation sets (ideally several thousand tokens per category) produce more reliable layer localization results.

To run a probing study, you need to extract the hidden states from a pre-trained model at every layer. With transformers from the Hugging Face library, this is straightforward. The critical keyword is output_hidden_states=True. When this is set, the model returns a tuple containing the hidden states at every layer, rather than only the final output. For a 12-layer BERT-base model, you get 13 tensors total: one for the embedding layer and one for each transformer block.

It is important to run the model in evaluation mode (model.eval()) and with torch.no_grad() to disable gradient computation. Since you are not fine-tuning the model, this is both faster and prevents any accidental weight updates. The representations you collect are the raw hidden states before any task-specific head, which is exactly what probing studies require: frozen, unmodified internal representations.

For sentence-level tasks (rather than token-level ones), you have additional aggregation choices. The [CLS] token representation is commonly used because BERT was trained to encode sentence-level information into it for the next-sentence prediction task. However, mean-pooling over all non-padding tokens often performs comparably or better on tasks that are not directly tied to the [CLS] training objective. The choice between CLS and mean-pooling can affect probe accuracy, so it is worth trying both and reporting results for both when presenting a probing study.

Another subtle choice is whether to extract representations before or after layer normalization. Most implementations return the post-normalization representations, which have been scaled to have consistent magnitude across the layer. This is usually the right choice for probing, since it ensures that probes at different layers receive comparably scaled input features. Probing pre-normalization representations would introduce confounds from scale differences across layers.

Transformers use subword tokenizers (WordPiece, BPE), which split words into pieces. The word "running" might become ["run", "##ning"]. If your probe targets word-level properties (like POS tags), you need to aggregate token-level representations. Common choices:

• First subword: Use the representation of the first subword token. Simple, fast, commonly used.
• Mean pooling: Average the representations of all subword tokens for a word. Tends to perform slightly better but requires token alignment.

The choice between these strategies matters more for morphologically rich languages, where a single word might be split into three or more subword pieces. In English, the difference is usually small, but in German or Finnish it can be substantial. For most English probing studies, the first-subword strategy is sufficient and is the standard baseline.

One subtle issue with subword alignment is that the tokenizer may insert special tokens ([CLS] at the start, [SEP] at the end) that shift all positions by 1. The Hugging Face word_ids() method returns None for these special tokens, making alignment straightforward: simply skip positions where word_id is None.

The aggregation question also matters for very long words. The word "nationalization" splits into ["national", "##ization"] and the two subword representations may encode different aspects of the word. The first subword "national" carries the stem meaning, while "##ization" carries the nominalization morphology. Mean-pooling integrates both signals, which can be beneficial for morphological probing tasks but may introduce noise for tasks where only the stem matters.

This section walks through a complete layer probing study on BERT. We will probe for part-of-speech tags across all 13 layers (embedding + 12 transformer layers) and visualize how accuracy varies with depth.

We start by installing the libraries we need and importing them.

Rather than downloading a large corpus, we create a controlled dataset of sentences with known POS tags. This keeps the example self-contained and fast to run.

The dataset is intentionally simple: 20 sentences, 8 coarse-grained POS categories, and a modest total of tokens. In a real probing study you would use a dataset like Penn Treebank or Universal Dependencies with tens of thousands of annotated tokens, but this synthetic dataset is sufficient to illustrate the methodology and produces a layer profile that mirrors the trends seen in larger-scale studies.

We now run each sentence through BERT and collect the hidden states at every layer. The key is to pass output_hidden_states=True and then align the subword token representations back to word-level tokens using the first-subword strategy.

We extracted representations for every word at every layer. Each word now has 13 representation vectors of 768 dimensions, one for each layer of BERT-base. The total data matrix has shape (total_words, 13, 768), which we will slice layer-by-layer when training probes.

With representations in hand, we train a logistic regression probe at each layer. We use a simple 80/20 train/test split.

Notice that we train a fresh logistic regression at each layer independently. The probes do not share weights and do not communicate with each other. This independence is essential: if the probes at different layers shared parameters, you could not attribute performance differences to the representations rather than to probe interactions.

The logistic regression uses L2 regularization by default (controlled by the C parameter, where smaller values mean stronger regularization). With C=1.0 and 768 input features, the probe has thousands of parameters relative to a few hundred training examples. This means regularization is doing meaningful work to prevent overfitting, and the accuracy numbers reflect useful information in the representations rather than noise memorization.

The accuracy profile is the central output of a layer probing study. Plotting it reveals which layers are most informative for POS tagging.

The curve typically shows the embedding layer (layer 0) with the lowest accuracy because it carries no contextual information. Accuracy rises through the early transformer layers as the attention mechanism begins to integrate local context, and the curve flattens as the representations converge to a stable encoding of syntactic categories.

The shape of this curve is informative in several ways. A steep rise followed by an early plateau suggests that the property is quickly saturated by local context, requiring little long-range integration. A gradual rise that continues to the final layers suggests the property benefits from global contextual information and is not easily captured from local patterns alone. A curve that peaks in the middle and then declines in the final layers is particularly interesting: it suggests that the final layers overwrite the representation in a way that makes the property less linearly accessible, which often happens for pre-trained models fine-tuned on downstream tasks that do not require the probed property.

We can also measure how much the representations change between consecutive layers.

To see how different linguistic properties are distributed across layers, you can probe for multiple tasks and overlay their accuracy curves. Here we simulate syntactic, semantic, and lexical probes to illustrate the typical pattern.

Overlaying multiple probe curves on the same axis is one of the most informative visualizations in layer probing research. When lexical probes plateau early and semantic probes continue rising, the divergence between the curves reveals the model's internal processing hierarchy directly. When two probes track closely, it suggests the corresponding properties are encoded at the same computational depth, which can point to shared processing mechanisms.

Task localization is a more targeted application of layer probing. Rather than asking "what is encoded at each layer?", it asks "at which specific layer(s) is the information most concentrated?" This matters for transfer learning, pruning, and deciding which intermediate representations to use for downstream tasks.

The simplest localization metric is the argmax of the probe accuracy curve: the layer that achieves the highest accuracy for a given task. But peak accuracy alone can be misleading. A more informative measure is the area under the probe curve (AUPC), which considers accuracy across all layers and rewards models where information is accessible from many layers (robust encoding) versus only one (fragile encoding).

Another useful measure is the layer of first saturation: the earliest layer at which the probe accuracy reaches, say, 95% of its eventual maximum. This identifies the minimum depth needed for a task, which has implications for efficient inference. If POS tagging saturates at layer 3 of a 12-layer model, you can potentially cut computation short for tasks that only require POS information.

The saturation analysis has motivated a line of research on early exit inference, where the model produces a prediction as soon as its representations at an intermediate layer are confident enough, without running all remaining layers. For tasks that saturate early, early exit can reduce computation by 50% or more with minimal accuracy loss. Layer probing thus has direct engineering applications beyond pure interpretability.

You can also compute a center of mass for the probe curve: a weighted average of layer indices, with probe accuracy as weights. This gives a single number summarizing how deep the information is located, on average. A center of mass near 3 means the property is mostly in early layers; a center near 10 means it requires deep processing. Comparing centers of mass across tasks reveals the relative depth of different types of linguistic knowledge.

Layer localization has direct consequences for how you should do transfer learning. The common practice of fine-tuning the entire model or taking the final layer's [CLS] token works well when your downstream task aligns with the information at the top layers. But for tasks that are primarily syntactic (like constituency parsing), you might get better performance, or faster convergence, by starting from a middle-layer representation.

This insight underlies layer selection in parameter-efficient fine-tuning. Adapter layers and LoRA modules are often inserted only at certain transformer layers. Understanding which layers encode task-relevant information helps you target the insertions where they matter most, rather than distributing them uniformly.

In practice, the question of which layer to use for a downstream task is often answered by validation performance rather than probing results. But probing provides a principled starting point: begin your search at the layer where probe accuracy peaks for the most closely related linguistic property. For a text classification task that relies primarily on sentiment, start from the upper layers. For a morphological analysis task, consider the lower layers. Probing reduces the search space from all layers to a small set of plausible candidates.

Another practical consideration is the interaction between layer selection and model fine-tuning. When you fine-tune a model on a downstream task, the weights of all layers shift. A probing result from the pre-trained model may not predict which layers will be most useful after fine-tuning, especially if the fine-tuning task requires substantially different representations from the pre-training objective. In these cases, probing the fine-tuned model (rather than only the pre-trained one) gives a more accurate picture of where the task-relevant information resides.

A powerful visualization is a heatmap of probe accuracy, where rows are tasks and columns are layers. This gives you an overview of the model's internal organization and lets you identify which layers are "specialists" (high accuracy on a narrow set of tasks) versus "generalists" (moderate accuracy on many tasks).

The heatmap format makes it easy to compare tasks at a glance. If you run this analysis on a real model with many probe tasks, you often see clear horizontal banding: tasks that require similar types of information cluster together and have similar heat distributions. Coreference and NER, for example, tend to have warm colors in the same upper-layer columns, while morphological inflection and POS tagging share warm colors in the same lower-layer columns.

Layer probing has been applied to many transformer architectures, revealing that the low-to-high hierarchy of linguistic abstraction is not unique to BERT. However, the precise mapping depends on the model's training objective, architecture, and size.

Encoder models like BERT and RoBERTa are trained with masked language modeling (MLM) on the full sentence. Because every layer can attend to both left and right context, the model can integrate bidirectional signals from the very first layer. This supports early syntactic integration: a token in layer 1 already "knows" what comes both before and after it, which helps establish syntactic category quickly.

Decoder-only models (GPT-style) use causal masking: each token can only attend to previous tokens. This changes the flow of information. The model cannot know what comes after a token until it has processed all preceding context. As a result, probing studies on GPT models often find that syntactic and semantic information develops more gradually and may not be as cleanly localized as in bidirectional models. The profiles tend to be smoother, with less pronounced peaks, because the model has to maintain open uncertainty about future context throughout processing.

An interesting consequence of causal masking is that the representation of a token at a given layer encodes different information depending on what position it occupies in the sequence. A token late in the sequence, after many preceding tokens have been processed, carries richer contextual information than the same token appearing at position 1. This position-dependent variation in representation quality is less pronounced in bidirectional models, which makes GPT-style models somewhat harder to probe cleanly.

Cross-attention in encoder-decoder models (like the original Transformer for translation) adds another dimension. The decoder layers must integrate both their own representations and the cross-attention output over the encoder's top-layer representations. This makes layer probing more complex for these architectures. Probing the encoder layers of a translation model reveals that they develop rich representations of the source language, while probing the decoder layers reveals how the model builds up the target language representation in a way that is conditioned on the encoded source.

Larger models tend to show more gradual, smoother accuracy profiles. In a 24-layer BERT-large versus 12-layer BERT-base, the same linguistic properties still tend to peak in the lower third, middle third, and upper third of the network, respectively. This suggests the functional organization scales with depth rather than being pinned to specific absolute layer numbers.

Larger models also tend to achieve higher peak probe accuracy overall. A 24-layer model's best POS probe outperforms a 12-layer model's best POS probe, partly because larger models have more capacity to encode linguistic information and partly because they are trained on more data. However, the relative improvement from adding depth is often more pronounced for higher-level semantic tasks than for surface syntactic tasks, which saturate quickly even in smaller models.

One particularly interesting scaling result concerns the distribution of linguistic knowledge across layers. In very large models (hundreds of layers or billions of parameters), the functional specialization becomes more pronounced rather than washing out. Individual layers in large models can show extremely sharp probing accuracy peaks, suggesting that certain computations become highly localized as model capacity grows. This is consistent with the view that large models can afford to dedicate specific computational resources to specific subtasks, rather than distributing processing diffusely.

The pre-training objective shapes what information is encoded at which layers. BERT's MLM objective requires the model to predict masked tokens using bidirectional context, which encourages the development of rich syntactic and semantic representations throughout. Models trained with next-sentence prediction or causal LM may develop representations with different distributions across layers.

RoBERTa, trained with more data, longer sequences, and no next-sentence prediction, shows similar but slightly sharper probing profiles than BERT. The representations at each layer tend to be more distinctive, making task localization cleaner. This suggests that the next-sentence prediction objective in original BERT may have introduced some noise that dilutes the quality of the layer-by-layer probing signal.

Instruction-tuned models and RLHF-finetuned variants show an interesting asymmetry: probing accuracy on linguistic tasks may decrease in upper layers relative to the base model. The fine-tuning process re-shapes the upper layers to produce task-appropriate outputs, which can overwrite some of the purely linguistic structure that was present in the pre-trained model. This means that if you want to probe for generic linguistic properties in a fine-tuned model, middle layers are often a safer choice than the final few layers. The middle layers tend to be less affected by instruction fine-tuning because the fine-tuning gradient signal is strongest at the output end of the network.

Contrastively trained models (like SimCSE or sentence-transformers) tell a different story. Because these models are trained to produce high-quality sentence embeddings, the final-layer representations tend to be more semantically structured than in BERT. Probing accuracy for semantic tasks like coreference or semantic textual similarity is often highest at the very top layer in contrastive models, rather than spreading across the upper third as in BERT. This reflects the training objective: a contrastive loss directly encourages the final representation to encode semantic similarity, concentrating semantic information at the output end.

Layer probing is powerful, but interpreting its results requires care. Several important limitations have been identified, and understanding them is as important as understanding the methodology itself.

Finding that a probe can decode property $P$ from layer $l$ does not mean layer $l$ uses property $P$ to make decisions. There is an important distinction between information being present (encoded in a representation) and being causally relevant (influencing outputs). A property could be encoded in the representation as a kind of side effect of other computations, without being directly used downstream.

Consider the analogy of a photograph of a person. The photograph encodes a great deal of information about the person (height, hair color, clothing, approximate age) that is not relevant to any particular task you might use the photograph for. A classifier trained to detect one feature (say, gender) from the photograph would succeed, but that does not mean the feature caused the photograph to exist or that it influences what any other classifier does with the photograph. Similarly, a linguistic property encoded in a BERT representation may be there incidentally, not because the model relies on it for its masked language modeling task.

This is why probing is often paired with other interpretability methods. Attention analysis can show where the model is looking. Gradient-based attribution can show which input tokens most influenced a decision. Activation Patching (which we will explore in the next chapter) can directly test whether manipulating a layer's representation changes model behavior. Probing alone provides a correlational, not causal, picture.

The causal gap is particularly important when drawing conclusions about model behavior. If you find that layer 8 encodes coreference information, you might conclude that the model "uses" coreference to resolve pronouns. But the model might be making those pronoun resolution decisions based on attention patterns that implement a completely different algorithm, and the coreference signal at layer 8 might just be a byproduct of that computation. Causal interventions (patching, ablations) are required to close this gap.

A probe with too much capacity can succeed by doing its own feature learning rather than reading off linearly encoded information. A very deep neural probe trained on BERT representations might achieve high accuracy on every task at every layer, not because BERT encodes everything everywhere, but because the probe can extract and combine information that is only implicitly available.

The solution is to use the simplest possible probe (linear, or at most a single hidden layer with strong regularization) and to compare against a control task with randomly shuffled labels. If accuracy on the shuffled-label task is significantly above chance, the probe has too much capacity and is memorizing rather than decoding.

The shuffled-label control is a critically important diagnostic. When you shuffle the labels in your probing dataset and train the same probe, you are creating a task that requires memorization rather than generalization. A well-behaved linear probe should achieve close to chance accuracy on shuffled labels (since there is no linear structure to exploit). If it achieves substantially higher accuracy, the probe is memorizing the training set, which means the accuracy numbers on the real task are also inflated by memorization.

A complementary control is the random model baseline: probe a randomly initialized (untrained) BERT model. The random model's representations have no linguistic content, so any probing accuracy above chance must come from the probe's ability to exploit statistical regularities in the dataset (like class imbalance) rather than from the representations. Comparing the trained BERT probing results to the random baseline gives a cleaner estimate of how much information is specifically contributed by pre-training.

Probe accuracy depends on the distribution and size of the training data. Small probing datasets may not have enough examples of rare categories to reliably measure accuracy. Imbalanced class distributions can cause the probe to default to the majority class. Any interpretation of layer probing results should account for the composition and size of the probe dataset.

One often-overlooked issue is the relationship between the probe dataset and the pre-training data. If the probing sentences come from the same domain as the pre-training corpus (e.g., news text), the model's representations may be especially well-tuned for that domain. Probing results from out-of-domain sentences (medical text, legal documents, social media) might tell a different story about which layers encode which properties, because the model has seen less training data from those domains.

Dataset construction choices can also introduce spurious correlations. If all the LOCATION entities in your NER probing dataset happen to appear at the end of sentences, and if position is weakly predictive of entity type, then a probe for LOCATION might succeed partly by detecting sentence position rather than entity-type information. Careful dataset construction, with cross-validation across different sentence positions and syntactic structures, is necessary to avoid these artifacts.

A critical subtlety is that BERT uses residual connections. The hidden state at layer $l+1$ is not computed from scratch; it is the sum of layer $l$'s output and the attention/FFN sub-network's output. This means information from early layers is directly propagated to all later layers. When a probe at layer 7 correctly identifies a POS tag, it might be reading off information that was computed at layer 2 and simply passed forward through residual connections.

Some researchers have proposed removing the residual by probing only the delta added by each layer (the FFN or attention output before it is added back). This gives a cleaner picture of what each layer contributes rather than what all preceding layers have accumulated.

The residual dependency problem makes it difficult to attribute specific computations to specific layers. If you observe that probe accuracy for POS tagging is high from layer 2 onward and stays high through layer 12, you cannot conclude that all those layers are computing POS-relevant information. It is equally consistent with the hypothesis that layer 2 computed POS information once, and layers 3-12 simply carried it forward in the residual stream without further modifying it. Probing the deltas rather than the full activations provides a way to test this hypothesis directly.

Another approach to this problem is ablation probing: after identifying a layer that encodes a property, remove that layer (by replacing its output with a zero tensor or a bypass) and measure whether downstream probe accuracy drops. If removing layer 2 causes POS probe accuracy to drop substantially at all subsequent layers, that confirms layer 2 as the primary site of POS computation rather than merely a conduit.

Beyond the methodological limitations discussed above, there is a more fundamental question about what probing tells us. Probing measures linear accessibility of information: whether a hyperplane in representation space separates the classes. But many computationally important forms of information might be encoded non-linearly, or distributed across multiple representations in a way that no single token's representation captures.

For instance, syntactic agreement between a subject and a verb requires relating two token representations. A probe at a single token cannot capture this relational information directly (though it might capture it indirectly if the subject's representation has been updated to reflect agreement requirements). Probing for relational properties typically requires probing over pairs of representations, which increases both the search space and the risk of the probe capacity problem.

Probing also says nothing about the mechanism by which the model uses the encoded information. Even if layer 8 perfectly encodes coreference clusters (high probe accuracy), you do not know whether the model computes coreference by attending to relevant tokens in earlier layers, by matching entity representations stored in FFN weights, or through some combination of both. Mechanistic interpretability tools, which we cover in subsequent chapters, are needed to answer mechanism questions that probing cannot address.

Layer probing provides a fine-grained view of how linguistic information is organized across the depth of a transformer model. By training simple classifiers at each layer, you can map out which layers are responsible for which properties and observe the continuous evolution of representations from surface-level to semantically rich.

The key findings from a decade of probing research are consistent across model families:

• Lower layers encode surface and morphological features, along with basic syntactic categories.
• Middle layers encode deeper syntactic structure, including phrase and dependency relations.
• Upper layers encode semantic content, including entity types, semantic roles, and coreference.

This stratified organization mirrors the classical NLP pipeline and suggests that pre-training on natural language induces a natural hierarchy of abstraction. The hierarchy is not incidental: it reflects the computational dependencies among linguistic properties. Syntax provides the scaffolding that makes semantic interpretation possible, and the model learns to build this scaffolding before applying it.

Beyond the basic accuracy-by-layer profile, the field has developed more sophisticated probing methods including structural probing (which tests geometric organization of syntactic relationships), information-theoretic probing (which measures mutual information between representations and labels), and minimum description length probing (which penalizes complex probes to avoid capacity confounds). These methods provide converging evidence for the same general hierarchy while addressing different methodological concerns.

Practical takeaways for practitioners include the following guidance. When selecting intermediate representations for a downstream task, consider the linguistic level of that task and target the corresponding layers. Layer probing results inform adapter placement in parameter-efficient fine-tuning, suggesting that adapters targeting the layers most relevant to the downstream task should be more efficient than uniform distribution. Probing accuracy is correlational, not causal. Pair it with activation patching (covered in the next chapter) or gradient-based methods for causal conclusions. Always use simple probes (linear or minimal MLP) and compare against shuffled-label controls to ensure results reflect the representation, not the probe's capacity. Finally, be aware that residual connections mean later-layer probes may be reading information computed many layers earlier; probing layer deltas provides a cleaner measure of each layer's individual contribution.

The next chapter on Activation Patching will take interpretability further by letting us observe what information is encoded at each layer and actively intervene on specific activations to test whether they causally determine model behavior.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about probing layers in transformer models.