The World-Model Qualification Test › Criterion 3: Counterfactual Validity

Action conditioning asks whether the model responds to actions. Counterfactual validity asks whether those responses remain trustworthy away from the training distribution, which is exactly where a curious planner spends its time.

Three practices make this criterion concrete.

Evaluate on off-policy splits. Hold out data collected under a different behavior policy than training used, and report error separately for on-policy and off-policy slices. A model whose off-policy error explodes has learned the logging policy, not the world.

Probe the action margins. Systematically vary the action input around states the agent cares about and inspect the predicted response surface. When the system has a known monotone actuation effect over the probed range, the learned response should have the correct sign and a plausible gain. Unexpected reversals, discontinuities, or extreme slopes are evidence against the model.

Track uncertainty, not just point predictions. A model that knows where its data ends can warn the planner. A model that extrapolates confidently converts ignorance into confidently wrong plans. Uncertainty-aware world models are a first-class topic in Part XII: Reliable World Models; for qualification purposes it is enough to demand that the model have some notion of applicability, even a crude one.

A few words on why each practice works. Off-policy splits are the cheapest honest instrument because they require no new data collection, only disciplined bookkeeping: tag every trajectory with the policy that produced it, and refuse to report a single pooled number. The action-margin probe is a sanity check on the model's internal physics rather than its fit: derivative signs and monotonicity are among the few things we know about real systems independently of data, so violations are unambiguous evidence of learned nonsense. The uncertainty requirement is subtler than the other two, because calibration itself is hard to verify, but even a crude applicability signal changes the planner's failure mode from silent confident error to detectable out-of-envelope queries, and that difference is often the difference between a recoverable mistake and a catastrophic one.

Counterfactual validity interacts dangerously with compounding. During a long imagined rollout, a planner explores ever-further corners of the model's state space, and small counterfactual errors redirect the search into regions where errors are larger still. This is the mechanism behind model exploitation: the planner finds actions that look brilliant under the model and are merely exploits of its extrapolation bugs. We will study defenses against exploitation, such as pessimism and ensembles, in the model-based reinforcement learning chapter of Part II.

It helps to picture exploitation as an adversarial game between two components of your own system. The planner is an optimizer: give it a flawed model and enough search budget, and it can discover regions where the model departs from reality. The model is often least reliable in precisely those poorly explored regions. Neither component need be broken; their combination is. Many defenses limit the optimizer's trust or exposure: ensembles reveal disagreement, pessimistic objectives penalize uncertainty, support penalties discourage unsupported searches, and shorter horizons bound how long errors can compound. We will return to this pattern throughout the book.