The World-Model Evaluation Ladder › Limitations and impact

The ladder is a diagnostic framework. It does not order models independently of a task, nor does it make the five protocols a strict staircase.

A model can perform well on a short control horizon and poorly on a long open-loop simulation. That is not a contradiction: feedback and planner inputs differ. Likewise, predicting a full state vector and controlling one task-relevant component can require different fidelity. These are reasons to record context and component-level diagnostics rather than infer an untested capability.

Statistical interpretation depends on the collection protocol. The 800-transition example uses independent Gaussian inputs and residuals, so the marginal MSE standard-error formula above applies. Real sequences may have correlated errors, non-Gaussian residuals, or variation across initial conditions and environments. A one-run comparison can be informative in a deterministic controlled experiment; claims about performance across episodes require appropriate replication and uncertainty estimates.

Use paired loss differences when models share test transitions, and choose the splitting unit to match the question. A random transition-row split can preserve dependencies across train and test or leak trajectory-specific information. It does not always do so: independently generated rows are a different protocol. If deployment concerns new initial conditions or environments, split and replicate at those levels. [Part XI, Ch 5: Datasets, Benchmarks, and Experimental Design] is planned to cover the protocol and power analysis.

The toy itself is deliberately limited. It has a scalar fully observed state, hand-specified candidates, Gaussian noise only in the one-step sample, and deterministic trajectory tests. The control experiment uses a bounded 201-point action grid, no state-safety constraints, and a one-step objective. It says nothing about the safety of an unconstrained deployment, nor about the quality of a multi-step planner or belief-state estimator.

Within those limits, the arithmetic shows why the tested protocols cannot be substituted for one another. The one-step winner differs from the constant-rollout and same-environment control winner. The alternating trajectory ranking changes between 40 and 500 steps. Changing the true state coefficient changes the control winner again. These are examples under declared conditions, not general architectural rankings.

For deployment, choose the diagnostics that match the proposed use and state the untested uses explicitly. Open-loop simulation needs appropriate horizons and path statistics. Policies learned through model rollouts need intervention and closed-loop tests under their action-selection procedure. Transfer needs a declared shift and an adaptation budget. A matched multi-rung comparison can reveal differences, but an irrelevant or unidentifiable rung should not be reported as certified.

This framework gives concrete uses to the distinctions developed in [Part II, Ch 6: Model-Based Reinforcement Learning], [Part III, Ch 2: Sufficient State and State Abstraction], and [Part VI, Ch 1: Predictive and Generative Objectives]. It separates representation quality, transition fidelity, and the way a planner uses predictions. Model exploitation is one failure mechanism to investigate rather than assume. It is a planned topic of [Part XII, Ch 1: Failure Modes and Model Exploitation].

The remaining chapters in this part are planned to specialize these questions: perceptual and generative evaluation for observation quality and scoring conventions; state, physics, and causal diagnostics for structured properties; planning and policy evaluation for decision outcomes; datasets and experimental design for reproducible comparisons; and interpretability and debugging for mechanisms behind failures.

For images, video, or point clouds, "closeness" itself requires care. Different valid images can have large pixel distances; identical point sets can have different orderings. Such metrics must state their representation and invariances. Those questions are planned topics of the next chapter, Perceptual and Generative Evaluation.