The World-Model Qualification Test › Why Open-Loop Accuracy Fools Us

In @ch-what-is-a-world-model we established the cast of characters: the environment evolves through transitions, the agent receives observations, chooses actions, and a world model is the agent's internal machinery for predicting how observations and rewards respond to actions. Everything in this chapter builds on that separation. The single most important consequence of it is this: the data an agent collects is generated by its own behavior, but the decisions it must make later live in situations its behavior creates. Evaluation procedures that ignore this gap systematically flatter bad models.

This circularity deserves a moment of careful attention, because it has no analogue in supervised learning on static datasets. When you train an image classifier, the test set is drawn from the same distribution as the training set by construction, and the classifier's predictions do not influence which images appear next. In sequential decision making, both guarantees vanish. The data distribution is a function of the behavior policy, and the deployment distribution is a function of the learned policy, so the two distributions are coupled through the very thing being learned. A model can be excellent at describing the world as the data-collection policy experienced it and simultaneously useless for describing the world as a new policy will create it. Every evaluation protocol that samples test cases from the logging distribution quietly assumes away this coupling, and thereby assumes away the hardest part of the problem.

The gap shows up in three distinct ways.

First, one-step error hides compounding. Most models are trained to predict one step ahead, where each prediction is graded against a ground-truth previous state. When the model is used for planning, however, its own outputs become its inputs. A 1% one-step error can become a useless prediction fifty steps later, because errors feed back into the model and grow. We saw the skeleton of this argument in Chapter 1 when our first world model drifted off course during long rollouts; here we will measure it precisely. The mechanism is easy to state but easy to underestimate: at each step of an imagined rollout, the model receives an input that is slightly wrong, produces an output that is therefore slightly more wrong, and passes that output forward. Under mild conditions the error grows geometrically rather than linearly, which means that horizon, not one-step accuracy, is often the binding constraint on usefulness. A planner that looks twelve steps ahead cares about a quantity that a one-step validation loss says nothing about.

Second, training distributions are not decision distributions. A model trained on logs of a careful human driver has seen gentle steering at moderate speeds. Ask it what happens during an emergency swerve and it is extrapolating. The model's accuracy on the logging distribution tells you almost nothing about its accuracy on the counterfactual actions a planner wants to explore. This is not a corner-case pathology; it is structural. Planners are, almost by definition, devices for finding actions that the behavior policy did not take, because if the best action were always the habitual one there would be nothing to plan. So the queries a planner directs at its world model are systematically concentrated exactly where the training data is thinnest. The better the optimizer inside the planner, the more aggressively it probes those thin regions, which is why improving the search algorithm can paradoxically make a model-based agent worse.

Third, a good predictor can still be a bad advisor. Prediction quality is measured against the truth; decision quality is measured against a reward. A model can be accurate everywhere the agent will go and still mislead the planner if its errors are largest exactly where the promising actions are, or if it is confident precisely where it should not be. To make this concrete, imagine two models with identical average error. One makes small uniform mistakes everywhere; the other is nearly perfect except that it systematically underestimates the cost of the single most attractive action sequence. Average error cannot distinguish them, but the second model is far more dangerous, because planners do not average over actions, they select the argmax. Decision quality depends on the error profile near the optimum, not the error profile overall, and these can be wildly different quantities.

These three gaps motivate the four criteria. State sufficiency and action conditioning concern what the model represents. Counterfactual validity concerns where its predictions hold. Decision usefulness concerns whether any of it matters when the model is placed inside an agent. Let us take them one at a time.