The World-Model Evaluation Ladder › The World-Model Evaluation Ladder

Suppose two dynamics models are evaluated on the same held-out robot transitions. One has lower one-step squared error. Is that enough to choose the model for a controller? The worked example below gives a concrete counterexample: the one-step winner is not the best tracker, and changing the environment changes the control ranking again. The example is synthetic; it is not a report of a deployed robot.

The difference comes from how a model is used. A one-step test supplies a real state as input. An open-loop simulator instead feeds predictions back into the model. A controller queries candidate actions, executes one in the real environment, and replans after observing the successor. These uses expose different errors.

For a model trained by one-step teacher forcing, the training pairs contain observed inputs and successors. Deployment need not use that same protocol: a rollout can visit predicted states, a planner can consider alternative actions, and a decision objective can weight errors differently. Other world models train on sequences or multi-step objectives, and many deployed controllers retain feedback. The distinction here is between evaluation protocols, not a universal account of world-model training.

Think of checking individual steps of a worked solution versus checking the assembled solution. Both checks are useful, but one does not replace the other. In a world model, recursion and action selection are part of that assembly.

The world-model evaluation ladder organizes five complementary questions.

• One-step prediction. Given a real transition $(s,a,s')$, how accurate is a point prediction or predictive distribution under the stated test law? In the point-prediction test here, every input is a real state.
• Open-loop rollout. From a common initial state and fixed actions, how far does a recursively predicted trajectory depart from the true trajectory, without observation corrections?
• Counterfactual intervention. How accurately does the model predict the response to changing the actions? Observational identification and unit-specific counterfactuals need assumptions beyond predictive accuracy.
• Closed-loop planning. How well does the model-planner combination act in the true environment when feedback is available?
• Transfer. Does that combination remain useful under a declared change in dynamics, observations, initial-state regime, objective, or constraints?

These are not strictly nested tests. A good result on one protocol does not certify another protocol with different inputs or objectives. A scalar score can be useful for a declared deployment objective. The ladder supplies a diagnostic vector alongside it, so that a scalar does not conceal untested uses.

Ranking models requires that context. A prediction winner, simulator winner, and control winner can be different models; they need not differ in every experiment.

We keep observation ($o$), state ($s$), action ($a$), model ($f_\theta$), planner, and objective separate. Unless stated otherwise, $s$ is the fully observed state of a controlled dynamical system, as in [Part I, Ch 2: What Is a World Model?] and [Part III, Ch 2: Sufficient State and State Abstraction]. Under partial observation, a belief or learned latent state must also be evaluated: an inaccurate upstream state estimate can affect the planner even when the transition model is accurate on its intended inputs. The ladder makes the object being scored explicit.