Interpretability and World-Model Debugging › Causal Interventions on Representations

Probe fitting and fixed-action rollout comparison can be observational diagnostics. Rollouts can also be part of an intervention experiment when actions or components are deliberately changed. A representation intervention changes a specified internal quantity and measures the response. It tests an internal computational effect under that protocol, not automatically a physical-world causal mechanism.

Geiger, Lu, Icard, and Potts applied causal-abstraction theory to neural-network analysis in 2021 (https://arxiv.org/abs/2106.02997). Their framework relates an interpretable high-level model to network computations through an alignment and interchange interventions; the broader theory predates this application.

The idea is to posit a high-level interpretable model and ask whether the network implements it. Formally, you specify an alignment between high-level variables and network representations. Then you run interchange interventions: you take a base input and a source input, run the network on both, and at a specified location you swap the representation from the source into the base. If the network's output then matches what the high-level model predicts under the corresponding counterfactual, the alignment is supported.

For each test, specify the high-level model, network site, alignment, and intervention rule. Candidate alignments may be proposed or found by search, as in Geiger and colleagues' study. Searching and validating on the same cases can favor a post-selected explanation; a prespecified held-out check helps test the selected alignment without turning that selection into independent evidence. An interchange result is a counterfactual computation under the stated intervention.

Geiger and colleagues demonstrated the method on MQNLI, a natural-language inference task. Their experiment is a text-classification study, not a robot-control study. The framework transfers, but the specific empirical findings do not. When you apply interchange interventions to a world model, you are making an analogy, and you should say so. The analogy is productive when the correspondences are made explicit and unproductive when they are not.

A well-known sequential-model example is the Othello-GPT work by Kenneth Li and colleagues (ICLR 2023, https://arxiv.org/abs/2210.13382). They trained a transformer to predict legal Othello moves from move sequences, then probed the internal representations for board state and intervened on them.

Two findings are commonly cited. A probe recovers board-state information from internal representations, and targeted representation edits change move predictions consistently with the tested board interpretation. These results concern the investigated models, inputs, and intervention sites.

This is a concrete, instructive case. But it is one synthetic task, and its scope is limited. Probe recovery of a board state is evidence of decodable board information, not evidence of general physical understanding. Intervention effects on move predictions are evidence that the model's predictions depend on the intervened representation, not evidence that the model plays optimally or reasons about the game the way a human does. And the board state is a fully observable, discrete, symbolic variable, which is a much friendlier setting than the continuous, partially observable, noisy state of a robot. The gap between the friendly setting and a real world-model is precisely where the analogy starts to strain, and it is the reason this chapter keeps returning to the point-mass toy rather than treating Othello-GPT as a template.

The TCAV-style directional test and the interchange intervention ask related but distinct questions. TCAV asks whether the output is sensitive to a direction. Interchange asks whether the output matches a counterfactual. Both are useful, and they can disagree. When they disagree, the disagreement itself is informative: a sensitivity test may fire on a direction that turns out to correspond to a nuisance, while a counterfactual test may fail on a direction that the network happens to respond to in ways the high-level model did not predict.

Hazineh, Zhang, and Chiu's 2023 follow-up (https://arxiv.org/abs/2310.07582) distinguishes player-relative mine/yours/empty labels from absolute black/white/empty labels. The player-relative board description is much more accessible to their linear probes, and intervention success varies with layer. Label semantics, readout family, and site therefore matter. These results do not by themselves rule out a coherent board representation or establish a unique encoding.

The lesson is general. "The model encodes X" is shorthand for a layered, probe-dependent, distribution-dependent statement. When you read or write such claims, unpack them. A useful unpacking template is: at which site, according to which probe family, on which distribution of inputs, under which split, and with what control?

A representation intervention is a strong tool, and it is easy to over-claim with it. Four limits are worth stating explicitly.

Off-manifold shifts. An arbitrary edit may produce an internal state absent from normal runs. Nearness of the base and source representations does not guarantee support of their combined state. Evaluate joint support and unintended coordinate changes where possible, and distinguish a valid computational intervention from a physically realizable state change.

Coordinate dependence. Invertible linear transformations preserve the existence of a linear readout: its weights can be transformed with the inverse map. A particular fitted regularized probe need not retain the same score under every transform. Nonlinear reparameterizations can change linear accessibility. Coordinate labels alone are not invariant semantics.

Nuisance confounding. A fitted concept direction may change nuisance attributes as well. Check those attributes and compare interventions designed to preserve them when the question requires isolation. An orthogonal direction or an unrelated sham is not automatically a matched nuisance control; explain which confounds each comparison addresses.

Recurrent state and dose. A partial edit can leave other memory variables inconsistent with an alternative input history. That limits a history-consistent physical interpretation, but does not invalidate a well-defined intervention on the internal computational graph. Specify all edited sites and times. A dose sweep reports the response to multiple edit magnitudes rather than treating one magnitude as a universal effect.

The laboratory uses a no-op sham, a dose sweep, and matched positions. It discusses the support limitations explicitly; position matching is not a completed joint-support check.