Video Generators as Candidate World Models › A Qualification Ladder for Video World Models

We propose a seven-rung ladder. Each rung has a ground truth, a baseline, a mode of false positive, and a characteristic way to interpret failure. Passing a higher rung provides stronger evidence for the capability tested on that task; passing only the early prediction rungs does not establish action-sensitive planning utility. None of these results, on its own, certifies or rules out the model's usefulness for every other task.

The ladder orders tests by what they ask of a model, from local prediction to decision-making. The middle rungs probe hidden-state tracking and action response; the later rungs probe uncertainty and closed-loop use. These are task-relative diagnostics, not a certification scheme: success on one rung neither guarantees the next nor makes every other rung necessary for a useful system. Report failures and evaluation conditions alongside successes.

What it tests. Given a held-out frame pair and the intervening action, does the model predict the next frame? This is the gentlest test.

• Ground truth. The recorded next frame from a held-out trajectory.
• Baseline. The trivial predictor that outputs the current frame (no motion). Also useful: the previous frame, or the mean frame over the dataset.
• False positive. A model that has memorized the training set will do unusually well on scenes that resemble training, without generalizing. Fix this by holding out scenes, not just timesteps, and by reporting per-scene performance.
• Failure interpretation. Beating persistence on a suitable held-out metric supports local predictive ability. Failing it calls for diagnosis of the data, metric, and task before making broader claims.

The one-step rung is easy to over-read. Much of a natural scene is predictable from the current frame alone, so a model can score well without tracking dynamics. Conversely, persistence is unusually strong when changes are small or a pixel metric penalizes valid stochastic alternatives. Treat this as a cheap diagnostic, then test the property the intended task needs.

What it tests. Condition on a real history and held-out action sequence, generate a $K$-frame continuation, and compare with reference outcomes. For recursive models this exposes compounding error; block or joint-clip models can fail differently.

• Ground truth. A held-out $K$-frame clip and its action trace.
• Baseline. A task-relevant latent dynamics model, such as those discussed in Part VIII, Chapter 2, and naive frame repetition.
• False positive. Models that generate plausible-looking but dynamics-wrong clips. Watch for slow motion, slowed time, or "frozen" backgrounds that hide drift in the salient object.
• Failure interpretation. Plot task-relevant state or event error against horizon, per scene, where reference state is available. Pixel error and perceptual scores are supplementary for stochastic futures. A steep curve limits the tested planning horizon; it does not settle whether the model has any world-model utility.

The key discipline on this rung is to report error as a function of horizon rather than as a single averaged value. A model whose error grows slowly with horizon is qualitatively different from one whose error saturates at a fixed offset, even if their average errors coincide. For a planner that scores trajectories out to horizon $K$, inspect error near $K$ as well as the average: late errors can change which action sequence looks best.

What it tests. Does the model maintain a belief about objects that are hidden but still exist?

• Ground truth. The true state of the occluded object while hidden, recorded in the simulator or estimated by external tracking from a second view.
• Baseline. A predictor that freezes the object's position at the moment it disappeared, and a predictor that uses the prior velocity to extrapolate.
• False positive. A model that appears to remember the object because in the held-out data the object always disappears while stationary. Fix by including occluded episodes with sustained motion.
• Failure interpretation. If an accessible state estimate or probe is less accurate than velocity extrapolation, that probe gives weak evidence of useful hidden-state tracking in this setting. Output-only generators may need a reappearance test instead. The corridor below illustrates visibility-stratified rollout error, not an isolated proof of memory.

Freezing the object at disappearance is a weak baseline. Velocity extrapolation is stronger, but beating it is not a definition of persistent state: the object's motion may itself be close to constant velocity, or the model's state may encode useful information that this particular probe misses. Use several motion regimes, evaluate after reappearance, and separate state-estimation error from rendering quality.

What it tests. Does the model's action channel actually control its predictions? See the earlier counterfactual note.

• Ground truth. Two real branches from a reset (a simulator episode with the same initial state and two different action traces).
• Baseline. A model that ignores the action channel; its two conditionings should produce statistically indistinguishable samples.
• False positive. A generator that is conditioned on the action in a correlated way because the training policy always applied the same action in that context. To expose this, evaluate on matched pairs where the two actions are both plausible under the training policy.
• Failure interpretation. Compare the predicted difference between branches with the reference difference. A near-zero predicted response is evidence of an action-channel failure only where the reference actions produce measurably different outcomes.

Matched initial states remove a major source of variation. For stochastic generators, compare branch distributions or couple samples with the same random seed; otherwise sampling noise can masquerade as an action effect. Correct branch differences support action sensitivity under the tested interventions, provided the reference reset and intervention are valid. They do not by themselves establish causal generalization to every policy.

What it tests. Can the model respond appropriately to specified scene or action changes, including shifts beyond well-covered training conditions? Recolor an object, move a distractor, change a container's shape, or apply an infrequent action where a reference response can be measured.

• Ground truth. The real simulator response to the intervention.
• Baseline. The base model without intervention on the same scene.
• False positive. Models that pass this test because the intervention is trivially local (recoloring the background does not change dynamics). The interesting interventions perturb dynamics-relevant attributes: mass, friction, contact shape.
• Failure interpretation. Compare the direction and magnitude of the predicted response with the reference. An unchanged prediction is a failure only if the intervention should change the measured outcome. This connects to the compositionality discussion in Part IV, Chapter 6.

This is a transfer test, but the input must contain enough information to identify the changed dynamics. Recoloring need not affect motion; a visible shape change might. Mass or friction cannot always be inferred from appearance alone. A model's response to a well-specified intervention is more informative than whether its output merely looks different.

What it tests. When the true future is ambiguous, does the model's sample distribution reflect that ambiguity, and when it is deterministic, does it concentrate?

• Ground truth. The empirical distribution of the next observation under the true stochastic dynamics. Often approximated by running the simulator many times.
• Baseline. A deterministic model that outputs the conditional mean, which will have under-dispersed samples.
• False positive. A model that is over-dispersed everywhere, including on deterministic transitions; this is easy to produce by cranking sampling temperature and looks like "handling uncertainty" without actually tracking it.
• Failure interpretation. Use reliability checks for defined events and proper scores where applicable: CRPS for scalar targets, or negative log-likelihood if the model exposes a tractable density. Report sharpness and predictive error separately. Part IV, Chapter 5 discusses broader uncertainty diagnostics.

Calibration asks whether stated probabilities agree with frequencies under a specified evaluation distribution; accuracy asks whether useful outcomes receive high probability or low error. A conditional-mean predictor can have low squared error yet miss a bimodal future. Broad samples may improve coverage while worsening sharpness. Neither property replaces the other, and empirical calibration requires repeated comparable conditions or carefully chosen event bins.

What it tests. Does using the model as part of a planner actually solve tasks? This is the only rung where the criterion is decision quality, not prediction quality.

• Ground truth. Task success rate and cost under a trusted reference on the same physical or simulated environment.
• Baseline. A strong model-free policy, a planning baseline using trusted dynamics where available, and a simple policy or random sequence, all under comparable observation and compute budgets.
• False positive. The model can pass this rung accidentally if the task is easy enough that a strong model-free baseline already succeeds, or if the evaluation environment leaks true state to the policy. The closed-loop eval must be a genuine planning loop with the model in the inner simulation step.
• Failure interpretation. A perceptually strong generator may still misrank actions. Inspect whether the planner exploits model error, as discussed in Part VI, Chapter 1, and retest the resulting actions in the reference environment. Report the action-distribution shift instead of assuming every closed-loop failure has this cause.

Closed-loop evaluation is decisive when the intended use is planning; other world models may be built for prediction or scientific analysis. A planner and model can jointly exploit errors in the learned simulator, so test chosen actions in the reference environment. When trusted dynamics are available, compare against an otherwise matched planner using them. Report uncertainty and policy shift rather than treating a single success rate as a complete diagnosis.