Probing Layers › Representation Evolution

One of the most conceptually important aspects of layer probing is what it reveals about how representations evolve through the network. A token's representation at layer 0 is purely lexical: it encodes the identity of the word. As information flows through successive layers, the representation undergoes a continuous transformation, accumulating contextual signals.

A useful mental model is to think of the representation $\mathbf{h}_i^{(l)}$ of token $i$ at layer $l$ as carrying increasingly rich information:

• Layer 0: "I am the token bank"
• Layer 3: "I am bank, a noun, and probably the subject of something"
• Layer 7: "I am bank, a noun, the subject of flooded, which suggests a geographical object"
• Layer 11: "I am bank, meaning a riverbank, playing the LOCATION semantic role in this event"

This evolution is not discrete or stepwise. Probing accuracy profiles are usually smooth functions of layer depth, suggesting gradual refinement rather than abrupt phase transitions. However, there are often local peaks and plateaus that indicate where specific types of computation are concentrated.

The continuous nature of this evolution has an important implication: no single layer "knows" the full answer. The model does not flip a switch at layer 8 and suddenly understand the meaning of the sentence. Instead, each layer adds a small increment of contextual understanding, like a growing blur of context sharpening into focus. This incremental enrichment is consistent with the residual architecture, where each layer adds a small correction to the incoming representation rather than recomputing it from scratch.

Researchers have also observed that representations from neighboring layers tend to be highly similar (cosine similarity often above 0.95) while representations from distant layers can diverge substantially. This suggests that each individual transformer block contributes a modest refinement, but the cumulative effect of many such refinements is dramatic. The embedding layer representation and the final layer representation of the same token often point in very different directions in the 768-dimensional space, even though consecutive layers look almost identical.

One intriguing finding concerns the way contextual information "flows" into a representation. Early layers, even with bidirectional attention, tend to integrate information primarily from nearby tokens. This is partly because attention weights in early layers are often more diffuse and locally biased. Upper layers can exhibit long-range attention patterns that integrate information from tokens many positions away. This means that the disambiguation of "bank" in a long sentence might require upper layers that can attend to a word ten positions back describing the setting as a river valley.

A complementary way to study representation evolution is to measure how much the representation of a token changes from one layer to the next. One approach uses cosine similarity:

$$\text{sim}(l) = \frac{1}{n} \sum_{i=1}^{n} \frac{\mathbf{h}_i^{(l)} \cdot \mathbf{h}_i^{(l-1)}}{\|\mathbf{h}_i^{(l)}\| \|\mathbf{h}_i^{(l-1)}\|}$$

where:

• $\mathbf{h}_i^{(l)}$: the representation of token $i$ at layer $l$
• $n$: the number of tokens in the sequence
• $\text{sim}(l)$: the average cosine similarity between consecutive layers $l$ and $l-1$

Low similarity means the representation changed a lot between those two layers. This can identify "active" layers where significant computation happens, as opposed to "pass-through" layers that barely modify their input. In BERT-base, researchers have found that the middle layers tend to show lower inter-layer similarity, suggesting they do more representational work than early or late layers.

Another measure is angular distance, the angle between consecutive layer representations:

$$\theta(l) = \arccos\!\left(\text{sim}(l)\right)$$

where $\theta(l) \in [0, \pi]$. A value near 0 means the representation barely changed; a value near $\pi/2$ means a 90-degree rotation, indicating substantial transformation.

The inter-layer similarity analysis complements probing by providing a different diagnostic lens. Probing tells you what is accessible from a given layer. Inter-layer similarity tells you where the most active computation is happening. Ideally, you would observe that layers with the most change in similarity also correspond to layers where new linguistic properties become accessible in probing. This correspondence is not always perfect, partly because the residual stream means changes in the delta do not necessarily translate to changes in the full representation, but it provides a useful cross-check on probing interpretations.

The residual connection architecture means that we can think of the model as maintaining a shared "residual stream" that all layers read from and write to. Each layer reads the current state of the stream, computes a small additive update (the output of the attention and FFN sub-operations), and writes that update back to the stream. The final representation at layer $L$ is the sum of all the delta updates from layers 0 through $L$.

This perspective has a direct implication for probing. When a probe at layer 7 correctly classifies a token's POS tag, it might be reading information that was computed at layer 2 and simply accumulated unchanged in the residual stream ever since. The probe cannot distinguish between information actively computed at layer 7 and information passively carried from layer 2. This is why some researchers have proposed probing the residuals rather than the full layer outputs: instead of probing $\mathbf{H}^{(l)}$, probe only the delta $\Delta^{(l)} = \mathbf{H}^{(l)} - \mathbf{H}^{(l-1)}$. The delta probe isolates what each layer contributed, rather than what all preceding layers accumulated.