Probing Classifiers › Structural Probing: Beyond Accuracy

Ordinary probing classifiers ask whether a discrete property (POS tag, dependency label) is encoded as a linear function of a single token's representation. Structural probing, introduced by Hewitt and Manning (2019), asks a more ambitious question: is the metric geometry of the representation space isomorphic to the metric geometry of syntactic parse trees?

The paper's core claim is striking. For any sentence, the syntactic parse tree defines a metric on word pairs: the tree distance between two words is the number of edges on the shortest path connecting them in the tree. Hewitt and Manning hypothesized that BERT's representations encode this metric structure as a global geometric property of the representation space rather than as isolated features for each word.

To test this, they learned a linear transformation $\mathbf{B}$ such that the squared Euclidean distance between transformed word representations correlates with parse tree distances:

$$d_{\text{tree}}(w_i, w_j)^2 \approx \left\| \mathbf{B}\mathbf{h}_i - \mathbf{B}\mathbf{h}_j \right\|^2$$

where:

• $d_{\text{tree}}(w_i, w_j)$ is the tree distance between words $w_i$ and $w_j$ in the gold parse tree
• $\mathbf{h}_i$ and $\mathbf{h}_j$ are their BERT representations
• $\mathbf{B}$ is a learned linear transformation (the structural probe)

The key architectural choice is that $\mathbf{B}$ is a shared transformation, applied identically to every word representation. It is a global metric space transformation rather than a per-word classifier. This is a much stronger structural claim than a standard probe: each word's representation encodes parse tree depth, and the entire parse tree is encoded as a metric geometry in representation space.

The transformation $\mathbf{B}$ projects representations into a lower-dimensional subspace (they use rank 64 out of 768 dimensions) in which tree distances are recovered. The fact that a 64-dimensional linear subspace of BERT's 768-dimensional space is sufficient to recover parse tree distances suggests that syntactic structure is encoded in a small, structured subspace of the representations, not diffusely spread across all dimensions.

This finding was influential for several reasons. It gave a geometric interpretation to what BERT learns: the model doesn't just classify words by POS, it organizes the representation space so that syntactically related words are geometrically close, in a way that exactly mirrors the parse tree structure. The structural probe also scales naturally to larger models and other architectures: any model that produces token-level representations can be probed for structural encoding.