Interpretability and World-Model Debugging › Probes and Concept Analysis

A useful question about a learned representation is whether a readout can recover a human-meaningful variable: position, velocity, or object identity, for example. A probe is an auxiliary model trained to predict a target from the representation. Successful held-out prediction supports decodability under that probe and evaluation protocol.

A linear probe tests affine readout performance; a polynomial or MLP probe tests a different function family with specified degree, architecture, and fitting procedure. Some nonlinear families contain the linear family, so their best attainable fit cannot be worse solely because their expressive capacity is larger. Fitted finite-data predictors can still perform worse through optimization, regularization, or overfitting. Report the family, distribution, and split when describing accessibility.

It helps to think of a probe as a measuring instrument. A thermometer does not create temperature; it reads it under a calibration and a range. A probe does not reveal the "true contents" of a representation; it reads whatever is linearly (or nonlinearly) extractable under a training protocol. This analogy is worth extending, because it captures the practical stakes. If you pick up a thermometer calibrated in a laboratory and use it outdoors in freezing weather, it may read nonsense, but that does not mean the temperature is undefined; it means the instrument was used outside its calibrated range. Similarly, a probe that reports low held-out accuracy on a distribution it was never calibrated for may be telling you almost nothing about the representation.

Three consequences follow.

First, successful decoding by a chosen probe is not sufficient to establish downstream use. It is not necessary either: a downstream computation may use a variable through a transformation that the tested probe cannot recover. Later, a nuisance-coordinate example shows the sufficient-direction failure directly: the readout recovers a supplied coordinate, while the controller ignores it.

Second, a failed fitted probe is not proof of information absence or an upper bound on the best readout in its family. Poor fitting, limited data, unsuitable regularization, distribution shift, or a restricted family can all reduce performance. The negative result describes that fitted instrument under that protocol. Testing another family or checking optimization can help distinguish these explanations.

Third, the probe's own capacity is a confound. A high-capacity probe can memorize. If you use a large probe and a small dataset, you can get high held-out accuracy on targets that are only spuriously correlated with the representation. This is why control tasks matter. Without a control, a probe result is compatible with two different stories: (a) the representation encodes the target, and (b) the probe is exploiting incidental structure to fit the target from data it should not be able to use. A matched control is how you begin to separate the two.

When you probe for a human-named concept, you are making a chain of assumptions. Each link can fail. Failures of assumptions needed for the intended interpretation can make the probe measure something different or limit that interpretation. The chain looks like this:

• You have a labeling procedure that assigns concept values to examples. The procedure may be noisy or may encode your own assumptions.
• The concept may be correlated with nuisance variables. If "has stripes" is correlated with "is a zebra," a probe for stripes may be learning zebra-ness.
• The probe has finite capacity, which constrains the readout functions it can represent. A restricted probe family may fail to recover an available concept; a high-capacity probe can fit incidental correlations or memorize, depending on the data and fitting protocol.

A concrete example: suppose you probe a driving world model for "is the traffic light red." If your dataset contains many more red lights at night than green lights, a probe may learn "is it night" and call it "red light." The probe reports high accuracy. The concept is not what you think it is. The high accuracy is real; the interpretation is broken. This is a nuisance-confounding failure, and it is easy to introduce accidentally because the nuisance correlation may be an artifact of how the data was collected rather than a property of the representation.

A control task helps assess the probe's capacity to learn a deliberately unrelated mapping while preserving specified properties of the real task. It does not automatically preserve every dependency or eliminate every concept correlate. Similar real-task and control-task scores can indicate low selectivity under this comparison; they do not prove that the concept is absent or that every real-task prediction uses spurious information.

John Hewitt and Percy Liang (2019, https://aclanthology.org/D19-1275/) assign independently sampled random control outputs to word types, consistently reusing each type's assignment across its tokens. This is not a permutation of linguistic types. Their selectivity measure is real-task accuracy minus control-task accuracy. Hypothetical scores of 0.95 and 0.55 give selectivity 0.40, whereas two scores of 0.95 give zero. The latter shows no advantage over that memorization control, not proof of concept absence.

A frame-level shuffled-target refit is different from a type-consistent control. Permuting training labels preserves their multiset and can change sample-to-label associations. An identity permutation, or swapping equal labels, leaves those observed associations unchanged. Even a nontrivial reassignment need not eliminate all accidental correlation. Its outcome depends on the split, shuffle, and fitting procedure. Use it as a randomized-label baseline, and repeat shuffles when their variability matters. To report Hewitt-and-Liang selectivity, construct the corresponding type-level task and disclose what is matched.

Probes are fitted models, and they inherit every discipline that fitted models require. It is tempting to skip this discipline because the probe is "just a diagnostic," but leakage can make its estimates misleading or compromise the intended held-out interpretation. A fitting-boundary violation does not guarantee that every prediction or score changes.

• For unseen-episode generalization, split by trajectory rather than frame. Samples from one episode share history and initial conditions. Mixing them across splits answers a within-episode question instead. Choose the grouping unit for the intended generalization target; held-out objects and held-out views likewise answer different questions.
• Fit data-adaptive preprocessing on training only in an inductive evaluation. Estimate standardization means and scales, PCA rotations, and other transformations learned from this experiment's data on training rows, then apply the frozen transform to validation and test. Fitting those statistics on the full dataset lets held-out information influence training. Fixed analytical transforms do not require fitting. Independently pretrained transforms require declared provenance and overlap checks, not compulsory refitting on this split. A disclosed transductive protocol permits different test-input access and answers a different evaluation question.
• Select penalties on validation, then freeze. Choose ridge penalties, probe architectures, and stopping criteria on a validation split. Once chosen, freeze the probe and evaluate once on the test split. Re-tuning after looking at test results invalidates the test split as a held-out estimate.
• Report the shuffle scope. Permuting training targets can change the fitting task. Permuting test targets can change the evaluation pairing without changing the already-fitted predictor, provided no refitting or test-driven selection follows. Identity permutations and swaps of equal targets leave the observed pairings unchanged. Specify which labels were randomized, whether assignments were consistent by type or episode, and how randomness was repeated.

We followed all four rules in the laboratory below. The split is by episode, preprocessing statistics come from training only, the ridge penalty is selected on validation, and the shuffled control shuffles training targets only.

A probe estimates a target from a representation. A sensitivity test instead chooses an output and asks how that output changes along a direction in representation space. TCAV, introduced by Been Kim and colleagues in 2018 (https://proceedings.mlr.press/v80/kim18d.html), is one such concept-based sensitivity protocol.

The TCAV recipe separates score construction from validation:

1. Define a concept by examples. Collect a set of inputs that exhibit the concept (for instance, images of striped textures) and a set that do not.
2. Fit a concept activation vector (CAV). A linear separator distinguishes concept examples from comparison examples at a chosen layer. Orient its normal toward the concept examples. This is a fitted direction, not a guarantee that moving along it increases the physical concept.
3. Measure directional sensitivity. Choose a class output and take its directional derivative along the CAV. For that class's examples, the original TCAV score is the fraction with a positive derivative; compute separate scores for separately chosen classes and sites.
4. Check whether the fitted direction yields a stable score. Kim and colleagues refit CAVs against multiple random contrast sets and apply a two-sided t-test against a score of 0.5, with a stated multiple-testing correction. Disclose the repeats, null hypothesis, and testing procedure. One raw score is not this validation, and a significant score still does not certify physical causality.

The TCAV score answers a specific question: how often does moving along the concept direction increase the output of interest? It is a quantitative, user-defined concept test. Note how the emphasis falls on "user-defined." The concept does not come from the model; it comes from examples you collected, a classifier you trained, and an output you chose.

A directional derivative measures local output sensitivity at the tested representation. A fitted direction can also track nuisance variation, so that sensitivity need not reflect the intended physical concept. A small derivative does not exclude finite-edit effects: the function $f(z)=z^2$ has derivative zero at $z=0$ but changes under a nonzero edit. Report whether the quantity is a derivative, an aggregate sign score, or a finite intervention effect.

It is worth stating the boundary conditions explicitly. Each of the following four distinctions separates a precise claim from a loose one, and each has a concrete failure mode in practice.

Sensitivity is not decodability. A representation can be decodable for a concept (a probe recovers it) while the output is insensitive to it (moving along the concept direction does not change the output). This is the "decodability without use" pattern. Conversely, the output can be sensitive to a direction that no probe recovers as a clean concept, because the sensitivity flows through a nonlinear interaction. Decodability and sensitivity therefore probe different things; neither finding alone establishes the other without additional assumptions about the concept, readout, output, and tested direction.

Decoding selected state variables with nonzero error does not prove Markov sufficiency. However, if every component of a sufficient finite-dimensional Markov state is exactly recoverable from the same representation, the component readouts jointly recover that state, and it is sufficient for predicting the next-state distribution given the action. The distinction is between partial or approximate recovery and exact recovery of a sufficient state, not a failure of exact component readouts to compose.

Directional sensitivity is not causal fidelity. The concept direction lives in representation space. Whether moving along it corresponds to a physically realizable change in the world depends on the data manifold, the encoder, and the task. An off-manifold latent shift can produce a large derivative without a physical interpretation. Latent proximity is a screening heuristic, not a guarantee of joint support or concept isolation. Check those properties separately when making a physical claim.

Intervention effects are not unique attribution. If replacing a latent variable changes the output, you have shown that the output depends on that variable under that intervention. You have not shown that the variable is the unique cause, that no other variable matters, or that the effect would hold under a different intervention protocol.

These four distinctions recur throughout the chapter. Keep them in mind as we move from probes to rollouts to interventions. Whenever you see a claim like "the model uses X" or "the model reasons about Y," try to identify which of the four distinctions the claim is glossing over.