The World-Model Evaluation Ladder

Suppose two dynamics models are evaluated on the same held-out robot transitions. One has lower one-step squared error. Is that enough to choose the model for a controller? The worked example below gives a concrete counterexample: the one-step winner is not the best tracker, and changing the environment changes the control ranking again. The example is synthetic; it is not a report of a deployed robot.

The difference comes from how a model is used. A one-step test supplies a real state as input. An open-loop simulator instead feeds predictions back into the model. A controller queries candidate actions, executes one in the real environment, and replans after observing the successor. These uses expose different errors.

For a model trained by one-step teacher forcing, the training pairs contain observed inputs and successors. Deployment need not use that same protocol: a rollout can visit predicted states, a planner can consider alternative actions, and a decision objective can weight errors differently. Other world models train on sequences or multi-step objectives, and many deployed controllers retain feedback. The distinction here is between evaluation protocols, not a universal account of world-model training.

Think of checking individual steps of a worked solution versus checking the assembled solution. Both checks are useful, but one does not replace the other. In a world model, recursion and action selection are part of that assembly.

The world-model evaluation ladder organizes five complementary questions.

• One-step prediction. Given a real transition $(s,a,s')$, how accurate is a point prediction or predictive distribution under the stated test law? In the point-prediction test here, every input is a real state.
• Open-loop rollout. From a common initial state and fixed actions, how far does a recursively predicted trajectory depart from the true trajectory, without observation corrections?
• Counterfactual intervention. How accurately does the model predict the response to changing the actions? Observational identification and unit-specific counterfactuals need assumptions beyond predictive accuracy.
• Closed-loop planning. How well does the model-planner combination act in the true environment when feedback is available?
• Transfer. Does that combination remain useful under a declared change in dynamics, observations, initial-state regime, objective, or constraints?

These are not strictly nested tests. A good result on one protocol does not certify another protocol with different inputs or objectives. A scalar score can be useful for a declared deployment objective. The ladder supplies a diagnostic vector alongside it, so that a scalar does not conceal untested uses.

Ranking models requires that context. A prediction winner, simulator winner, and control winner can be different models; they need not differ in every experiment.

We keep observation ($o$), state ($s$), action ($a$), model ($f_\theta$), planner, and objective separate. Unless stated otherwise, $s$ is the fully observed state of a controlled dynamical system, as in [Part I, Ch 2: What Is a World Model?] and [Part III, Ch 2: Sufficient State and State Abstraction]. Under partial observation, a belief or learned latent state must also be evaluated: an inaccurate upstream state estimate can affect the planner even when the transition model is accurate on its intended inputs. The ladder makes the object being scored explicit.

The one-step test scores a prediction at an observed input without feeding the model's previous prediction back into it. That makes it convenient for regression-style models, but the resulting score does not test recursive simulation or action selection. Teacher forcing also does not make transitions statistically independent; neighboring observations can remain correlated.

Let $T(ds'\mid s,a)$ be a controlled Markov transition kernel. It returns a probability law over successors, not a successor state vector. Let $D$ be a common evaluation distribution over $(s,a)$, and let $m(s,a)=\mathbb{E}_T[s'\mid s,a]$ be the conditional mean. For a deterministic point predictor $f$ with finite squared risk, assume the successor and predictor have finite second moments under this joint law. Then

The first term measures conditional-mean prediction error. The second averages conditional squared noise over the same evaluation law. It depends on $T$ and $D$, but not on the deterministic point predictor. To obtain the identity, write $f-s'=(f-m)+(m-s')$ and expand the squared norm. The conditional expectation of the cross term is zero because $\mathbb{E}_T[s'-m\mid s,a]=0$. Noise is already part of $L_1$; it is not added again to an expected squared error against sampled successors.

Where:

• $s$ and $a$: the joint state-action input drawn from $D$;
• $s'$: the sampled successor drawn from $T(\cdot\mid s,a)$;
• $m(s,a)$: the conditional mean successor;
• $f(s,a)$: a deterministic point prediction;
• $\sigma^2_{T,D}$: the average conditional squared noise norm.

Training data $\mathcal D=\{(s_i,a_i,s'_i)\}_{i=1}^N$ may be collected by a behavior policy and a state-estimation pipeline. The evaluation law can be chosen to match that collection regime or to represent a different use. The conditional-mean decomposition applies to the selected law; it does not establish that this law covers deployment.

On $M$ held-out transitions the empirical score is

Here $j$ indexes the evaluation transitions, $M$ is their count, and each $s'_j$ is the observed successor of input $(s_j,a_j)$. With independent evaluation draws from the specified joint law, conditional on a fixed fitted predictor, this sample average is unbiased for its population risk. Disjoint row indices alone do not establish independence from the fitted predictor or independence among evaluation rows. For trajectory data, the sampling and splitting unit needs to be stated.

Three quantities help interpret the score. Let $\mathcal F$ be a fixed hypothesis class, let $f^*_{\mathcal F}$ minimize $L_1$ over that class when such a minimizer exists, and let $\hat f\in\mathcal F$ be the fitted model.

• Approximation error is $L_1(f^*_{\mathcal F})-\sigma^2_{T,D}$, the best conditional-mean error available in the class.
• Fitting excess is $L_1(\hat f)-L_1(f^*_{\mathcal F})$. Finite-data estimation and imperfect optimization can both contribute to it.
• Evaluation error is $\widehat L_1(\hat f)-L_1(\hat f)$, a signed discrepancy due to the evaluation sample.

These definitions give an exact bookkeeping identity when the minimizer exists:

The final bracket can be negative; the first two excess terms cannot. More training data does not remove approximation error in a fixed misspecified class. Under suitable sampling, identifiability, and consistency assumptions it can reduce estimation excess over the best member, even when the class is misspecified.

For example, let $Y=X$ with $\mathbb E[X]=0$ and $\operatorname{Var}(X)=1$, but allow only constant predictors $f_c(x)=c$. The best constant is zero and its approximation risk is one. Fitting the sample mean on $N$ independent draws gives expected excess risk $1/N$. The approximation error remains while estimation excess vanishes. There is no universal rule that greater capacity always increases estimation error; that tradeoff depends on the class, algorithm, regularization, and data.

For a comparison on common evaluation transitions, uncertainty belongs to the paired loss differences. The marginal standard error of one model's MSE is not the standard error of the difference. A numeric gap such as $0.005$ cannot be judged from $M=800$ alone.

Finally, averaging signed prediction errors is not the same as averaging squared errors. Positive and negative mistakes do not cancel in MSE. A conditional mean can be the optimal point predictor yet fail to represent multimodal futures needed by a particular planner. That is a reason to evaluate a predictive distribution or the downstream task separately, not to claim that smoothing is automatically low-error.

A small two-component oscillator makes the distinction concrete without copying the answer into the predictor. Its true state is $z_t=(x_t,v_t)$, and its transition rotates that vector by $\omega=2\pi/20$. The same misspecified model is used in both tests: it rotates by the correct angle but expands the state by a factor of $1.02$. Teacher forcing resets each input to the true state; open-loop rollout feeds the model's previous prediction back into itself.

Three confounds deserve special attention.

1. Distribution coverage. The evaluation law $D$ need not match the state-action occupancy induced by deployment. If data cover only near-upright robot states, predictions far outside that region are extrapolations unless other data or structural constraints inform them. A state absent from a finite sample is not necessarily outside the support of the population distribution.
2. Stochasticity. Point-prediction MSE and distributional scores answer different questions. A probabilistic model can be scored by its mean for MSE, or by a suitable proper scoring rule for its distribution. Comparing a random sample from one model with a point estimate from another changes the scoring convention. Uncertainty-aware planning needs separate tests. [Perceptual and Generative Evaluation] is planned to examine observation-quality scoring, not certify a planning system.
3. Observation error. A state estimated from an image or point cloud can be inaccurate before the dynamics predictor receives it. Separating perception and transition errors needs component-specific measurements, identifiable model/noise assumptions, or interventions. Aggregate prediction error alone does not distinguish those components. Their units, directions, dependence, and propagation matter; two nominal error magnitudes alone do not determine the combined prediction error.

A one-step result is therefore a distribution-specific check. It can be decisive if that is the intended task. It cannot, by itself, establish rollout quality, intervention accuracy, or useful control.

An open-loop rollout feeds predicted states back into the model while using a fixed action sequence. This tests the composed transition map, including how errors at one input affect later predictions. It does not imply that the one-step evaluation samples were independent.

First distinguish two environments. In a stochastic environment, $s_{t+1}\sim T(\cdot\mid s_t,a_t)$. For the pathwise bound below, restrict attention to deterministic dynamics with a state-valued map $F$:

Where:

• $F$: deterministic true dynamics, returning a successor state;
• $T$: a transition probability kernel; in this deterministic special case it places all probability at $F(s,a)$;
• $s_t$ and $\hat s_t$: the true and predicted states;
• $a_t$: the common action at step $t$;
• $H$: the positive integer rollout horizon.

Both branches use the same initial state and action sequence $a_{0:H-1}$, with no observation corrections during prediction. Trajectory RMSE and endpoint error summarize different parts of the path:

The RMSE averages the $H$ successor states, excluding the common zero-error initial state. Endpoint error scores only the final state. Neither dominates as a metric: an application that needs the whole path may need intermediate errors, while terminal decisions may prioritize the endpoint. For stochastic rollouts, state whether the score compares distributions, expected quantities, or coupled sampled paths; unrelated noise draws can contribute to a pathwise difference.

To derive an error envelope, let $d_t=\|\hat s_t-s_t\|$ and assume:

• common fixed actions and a common initial state;
• a uniform $L\geq0$ satisfying $\|f_\theta(x,a)-f_\theta(y,a)\|\leq L\|x-y\|$ for every compared true/predicted input pair and action along the trajectories;
• a uniform local error bound $\|f_\theta(s,a)-F(s,a)\|\leq e_1$ on every true input encountered.

A sufficient stronger condition is that both bounds hold throughout a reachable region containing those trajectories. Add and subtract $f_\theta(s_t,a_t)$, then use the triangle inequality:

Here $d_0=0$, $L$ bounds amplification in the stated norm, and $e_1$ bounds local transition error. Iterating this inequality yields

These are upper envelopes, not observed growth laws. If the same uniform assumptions hold for every horizon and $L<1$, the envelope does not exceed $e_1/(1-L)$. At $L=1$ it is linear, and at $L>1$ it is exponential. Actual errors can remain zero even under a loose bound with $L>1$.

The distinction matters in three ways.

• Stability: one-step average loss does not measure the amplification of perturbations. For differentiable maps, a suitable uniform bound on the induced Jacobian norm can establish the Lipschitz condition on an appropriate region. A spectral radius at one state is not a general bound on products of changing Jacobians.
• Horizon: report the horizon and the trajectory statistic together. The same models can change order as the horizon changes.
• Ranking: similar held-out average risks do not determine rollout ranking or supply the uniform $e_1$ used above. A contractive model can sometimes outperform a less biased expansive model in rollout, but neither stability nor local accuracy alone fixes the ranking.

The following plot holds $e_1=0.01$ fixed and draws the envelope for three chosen values of $L$. It plots the bound, not measured trajectory errors.

The envelope assumes accuracy at the inputs reached by the rollout. A held-out average error under $D$ does not establish that condition. Predicted trajectories may enter regions with few observed samples, or regions outside the support of $D$ altogether. These are different cases: a finite sample can omit inputs that still have positive population probability.

This mismatch between observed training inputs and model-generated inputs is one mechanism behind compounding-error problems. It motivates training methods that explicitly confront model-generated states, such as Talvitie's self-correcting models. The paper analyzes that mechanism under its stated assumptions; it does not imply that every high-capacity model fails at long horizons. [Part VII, Ch 4: Policies Learned in Imagination] is planned to cover the planning use of such rollouts.

The intervention rung asks how predictions change when the action protocol changes. A controlled simulator comparison and an observationally identified causal effect are different ways of obtaining an answer. Predictive accuracy on collected transitions is not enough to establish the latter.

For a deterministic environment $F$, fix an initial state $s_0$ and two action sequences, baseline $a_{0:H-1}$ and intervention $b_{0:H-1}$. Run both sequences from that common state in the true and model systems. The final response is

The response error is $\|\Delta_H^{\mathrm{model}}-\Delta_H^{\mathrm{true}}\|$. Superscripts identify the action branch; the two branches share $s_0$. Changing only actions isolates their effect within this deterministic experiment. If the two model branch errors have an equal additive component, that component cancels in the response difference. Unequal or action-dependent errors need not cancel.

In a stochastic system, a transition kernel specifies each branch's marginal law, not how their exogenous randomness is paired. A unit-specific counterfactual comparison needs that additional causal structure. For example, two branches can each be Bernoulli$(1/2)$ while sharing the same outcome, taking complementary outcomes, or using independent outcomes. Their marginal means coincide, but their paired differences have different distributions. A structural transition $s_{t+1}=F(s_t,a_t,U_t)$ can specify a shared sequence $U_t$ across the branches; a kernel alone does not supply it. Pearl's structural-counterfactual account makes this dependence on background variables explicit.

Alternatively, define an average intervention response as $\mathbb E[s_H^b]-\mathbb E[s_H^a]$. That difference of marginal expectations does not require a cross-branch noise coupling, though estimating it from observational data still requires identification assumptions. Neither definition should be silently substituted for the other.

For nonparametric adjustment-based observational identification, specify a causally adequate pre-action state, consistency, conditional exchangeability (no unmeasured action-outcome confounding), and positivity for the actions being compared. Multi-step policies need corresponding sequential conditions. State-dependent action selection is not automatically confounding after conditioning on an adequate state. A deterministic behavior policy can nevertheless leave alternative actions unsupported. Extrapolating their effects requires additional structural assumptions. The roles of consistency, exchangeability, and positivity are developed in Hernán and Robins' Causal Inference: What If.

Two practical distinctions help interpret the score.

• Data and assumptions: unsupported action effects cannot be recovered from observational accuracy alone. Additional experiments, structural knowledge, or scientific priors can constrain them. A poor response result may reflect data coverage, incorrect assumptions, or model error; changing architecture alone does not establish identification. [Part IV, Ch 3: Causal Models and Interventions] treats these choices.
• Response features: sign, magnitude, and timing can matter differently to a controller. A final-state norm does not separately diagnose all three. Report the response itself alongside a scalar error when these distinctions matter.

A successful response test covers the specified states and interventions. It does not guarantee useful optimization gradients, safe control, or accuracy on all alternative actions. Those need further tests.

Closed-loop planning evaluates the model as part of a decision system. At each step the agent observes the real state, queries the model to choose an action, executes that action in the true environment, and observes again. For a deterministic environment the resulting state is $s_{t+1}=F(s_t,a_t)$; with stochastic dynamics it is sampled from $T(\cdot\mid s_t,a_t)$.

Re-observation prevents a predicted state from directly replacing the real next state. It does not sever the effect of model error on the trajectory: predictions influence the action, and the action influences future real states. Feedback can correct some errors, but an unstable or poorly chosen controller can still fail.

Small prediction perturbations need not cause small action changes. Consider actions $-1$ and $+1$, true next state $F(0,a)=a$, and target zero. An exact predictor gives equal squared costs, so an optimizer with a tie rule can choose $-1$. Perturb only the predicted $+1$ successor from $1$ to $1-\epsilon$, with $0<\epsilon<1$. The prediction changes by only $\epsilon$, but the unique preferred action is now $+1$, changing the real successor by two. Smooth action sensitivity or stable correction requires additional conditions on the optimizer, objective, constraints, and closed-loop dynamics.

Let $J(\pi)$ be expected cumulative cost in the true environment, with a common objective, initial-state distribution, noise law, horizon, and policy class $\Pi$. If $\pi_\theta\in\Pi$, define

Assume the costs and infimum are finite. If an optimal policy $\pi_{\mathrm{opt}}$ attains that infimum, the gap is $J(\pi_\theta)-J(\pi_{\mathrm{opt}})$. For rewards being maximized, use $\sup_{\pi\in\Pi}J_{\mathrm{reward}}(\pi) -J_{\mathrm{reward}}(\pi_\theta)$ instead. The sign depends on whether the objective is a cost or a reward.

Model approximation, finite-data estimation, planner suboptimality, and distribution mismatch are diagnostic categories, not universally independent additive quantities. An exact decomposition would need defined intermediate models and policies. We use the categories to investigate the observed gap.

A planner can exploit model errors by selecting actions whose predicted cost is favorable but whose real cost is not. This is one possible failure mechanism of optimizing through an inaccurate model. Greater search capacity does not invariably worsen performance: it can also find better actions. The direction depends on the errors, objective, constraints, and search procedure.

Three practical points follow.

• Feedback and rollout test different uses: replanning from observations can compensate for recursive model drift. Good tracking in that setting does not certify a long open-loop simulation.
• Report the whole decision system: the result belongs to the model, planner, objective, constraints, observation convention, and compute budget together. Changing any of them can change the ranking.
• Separate objective validity from decision optimization: a model can predict transitions accurately while the controller optimizes the wrong objective for deployment. State the intended objective and the scored objective before attributing a decision failure to dynamics.

The worked example uses observed states and a one-step grid search. It does not test belief-state inference, long-horizon planning, or safety.

Transfer repeats an evaluation under a specified change. Several changes should be distinguished.

• A dynamics shift replaces $T$ by a different transition law $T'$ in the queried region. A predictor accurate for $T$ can become inaccurate there.
• An initial-state shift changes occupancy and can expose poorly covered regions without changing the transition law.
• A changed objective or action constraint changes which decisions are useful even when the dynamics predictor remains correct.
• An observation shift can change the state or belief supplied to the controller while the physical transition law stays fixed.

Fitting data from $T$ alone establishes neither exact correctness under $T$ nor inevitable failure under $T'$. State what changed, what remains fixed, and whether retraining, adaptation, or extra observations are allowed. The experiment below changes only one true dynamics coefficient and keeps the candidate models, controller, target, initial state, and action grid fixed.

A scientific prior can help if it encodes structure that persists under the chosen shift; it can hurt if that structure changes. Better in-distribution fit does not universally imply worse transfer, and structural bias does not universally imply better transfer. [Part XII, Ch 2: Robustness, Calibration, and Safe Control] is planned to examine these questions more systematically.

Observation quality remains a separate part of the system, as discussed in [Part III, Ch 1: Observations, Sensors, and Multimodal Data]. Controlled changes to the perception pipeline can help distinguish observation failures from transition failures. The next chapters are planned to cover state and physics diagnostics.

The example below is deliberately misspecified and small, so that everything is on the table. It is a compact illustration, not evidence that any particular architecture is inferior.

We use a fully observable linear system with a deterministic state-coupling structure plus additive Gaussian process noise:

with $a^\star = 0.95$, $b^\star = 0.50$, and $\varepsilon_t \sim \mathcal{N}(0, 0.02^2)$ i.i.d. across $t$.

Where:

• $s_t$: the scalar state at time $t$

• $u_t$: the scalar control action at time $t$

• $a^\star$: the true state-coupling coefficient

• $b^\star$: the true action gain

• $\varepsilon_t$: i.i.d. Gaussian process noise with standard deviation $0.02$

• State: scalar $s_t \in \mathbb{R}$ (fully observed).

• Action: scalar $u_t \in \mathbb{R}$, with a hard bound $|u_t| \le 0.5$ for the planning tests.

• Planning objective: squared predicted next-state distance to $s^\star=1$, with no action penalty.

• Control score: mean absolute target error over true states 31–40. This diagnostic is not the cumulative squared-cost gap $G$ defined above.

• Episode horizon: 40 steps for rollouts, 40 steps for closed loop.

• Data policy: for one-step evaluation, actions are drawn i.i.d. from $\mathcal{N}(0, 1)$. For rollout evaluation, we use the fixed sequence we specify in each rung.

• Evaluation data: 800 independent synthetic transitions. The candidates are hand specified before sampling; no training set or fitted-model claim is involved.

• Noise regime: Gaussian process noise is present in the one-step sample only. Rollout, intervention, and control tests use the noiseless transition, so their errors measure misspecification rather than different noise draws.

• Model state: the Delayed candidate also carries the previous action. Its recurrent state is $(\hat{s}_t,u_{t-1})$, initialized with $u_{-1}=0$ in rollout and control tests.

We compare three candidate models of the form

where:

• $f_{\text m}$: the model's next-state prediction;
• $x_t$: the input state, true $s_t$ for teacher forcing and replanning, predicted $\hat s_t$ for an open-loop rollout;
• $p_t$: the previous applied action, $u_{t-1}$, with $u_{-1}=0$ for the trajectory experiments;
• $a_{\text{m}}$: the model's state-coupling coefficient
• $b_{\text{m}}$: the model's instantaneous action gain
• $c_{\text{m}}$: the model's action-lag coefficient, capturing the influence of the previous action

The following candidates are chosen to expose different failure modes.

• Aggressive ($a_{\text{m}} = 1.02$): captures the state coupling closely but overshoots it into instability.
• Conservative ($a_{\text{m}} = 0.80$): underestimates the state coupling.
• Delayed ($a_{\text{m}} = 0.95$, $c_{\text{m}} = 0.20$): gets the state coupling right but adds a spurious action lag.

These are hand-constructed misspecifications, not fitted solutions. Our goal is not to argue that any particular model class fails; it is to show that each rung rewards different inductive biases and that a lower rung does not by itself guarantee a higher one without additional linking assumptions. Notice that each model is a plausible way to be wrong about the same system. The Aggressive model is approximately correct as a one-step predictor because its coefficient error is small; the Conservative model trades a larger bias for more conservative behavior; the Delayed model gets the coupling right but invents a spurious dependence on the previous action. These hand-selected deviations let us isolate what happens when different coefficient errors are pushed through the evaluation tests.

The pattern is exactly what the ladder is designed to expose: different rungs can favour different winners.

• One-step MSE favors Aggressive on the stated independent Gaussian inputs.
• Constant-input rollout favors Delayed at the specified 40-step horizon, despite its incorrect gain.
• The response-difference metric favors Delayed for this baseline/intervention pair. The additional alternating-trajectory test is not the same metric.
• Same-environment grid control also favors Delayed; the example does not show a winner reversal between that rollout and that controller.
• Transfer favors Conservative because its coefficients match the changed environment more closely. This is no general guarantee of robust transfer.

These rankings are conditional on the test distribution, action sequence, horizon, initial state, and scoring rule. A scalar chosen for a declared deployment objective can be useful. Report that choice explicitly, and retain the other diagnostics when they describe additional intended uses.

The ladder is a diagnostic framework. It does not order models independently of a task, nor does it make the five protocols a strict staircase.

A model can perform well on a short control horizon and poorly on a long open-loop simulation. That is not a contradiction: feedback and planner inputs differ. Likewise, predicting a full state vector and controlling one task-relevant component can require different fidelity. These are reasons to record context and component-level diagnostics rather than infer an untested capability.

Statistical interpretation depends on the collection protocol. The 800-transition example uses independent Gaussian inputs and residuals, so the marginal MSE standard-error formula above applies. Real sequences may have correlated errors, non-Gaussian residuals, or variation across initial conditions and environments. A one-run comparison can be informative in a deterministic controlled experiment; claims about performance across episodes require appropriate replication and uncertainty estimates.

Use paired loss differences when models share test transitions, and choose the splitting unit to match the question. A random transition-row split can preserve dependencies across train and test or leak trajectory-specific information. It does not always do so: independently generated rows are a different protocol. If deployment concerns new initial conditions or environments, split and replicate at those levels. [Part XI, Ch 5: Datasets, Benchmarks, and Experimental Design] is planned to cover the protocol and power analysis.

The toy itself is deliberately limited. It has a scalar fully observed state, hand-specified candidates, Gaussian noise only in the one-step sample, and deterministic trajectory tests. The control experiment uses a bounded 201-point action grid, no state-safety constraints, and a one-step objective. It says nothing about the safety of an unconstrained deployment, nor about the quality of a multi-step planner or belief-state estimator.

Within those limits, the arithmetic shows why the tested protocols cannot be substituted for one another. The one-step winner differs from the constant-rollout and same-environment control winner. The alternating trajectory ranking changes between 40 and 500 steps. Changing the true state coefficient changes the control winner again. These are examples under declared conditions, not general architectural rankings.

For deployment, choose the diagnostics that match the proposed use and state the untested uses explicitly. Open-loop simulation needs appropriate horizons and path statistics. Policies learned through model rollouts need intervention and closed-loop tests under their action-selection procedure. Transfer needs a declared shift and an adaptation budget. A matched multi-rung comparison can reveal differences, but an irrelevant or unidentifiable rung should not be reported as certified.

This framework gives concrete uses to the distinctions developed in [Part II, Ch 6: Model-Based Reinforcement Learning], [Part III, Ch 2: Sufficient State and State Abstraction], and [Part VI, Ch 1: Predictive and Generative Objectives]. It separates representation quality, transition fidelity, and the way a planner uses predictions. Model exploitation is one failure mechanism to investigate rather than assume. It is a planned topic of [Part XII, Ch 1: Failure Modes and Model Exploitation].

The remaining chapters in this part are planned to specialize these questions: perceptual and generative evaluation for observation quality and scoring conventions; state, physics, and causal diagnostics for structured properties; planning and policy evaluation for decision outcomes; datasets and experimental design for reproducible comparisons; and interpretability and debugging for mechanisms behind failures.

For images, video, or point clouds, "closeness" itself requires care. Different valid images can have large pixel distances; identical point sets can have different orderings. Such metrics must state their representation and invariances. Those questions are planned topics of the next chapter, Perceptual and Generative Evaluation.

• The five protocols are one-step prediction, open-loop rollout, intervention response, closed-loop planning, and transfer. They are complementary questions, not strictly ordered guarantees.
• One-step squared risk under a common finite-moment evaluation law splits into conditional-mean error and average conditional noise. A held-out estimate needs a declared sampling protocol and uncertainty calculation.
• The deterministic rollout bound requires uniform reachable-input accuracy and Lipschitz conditions, common actions, a common initial state, and no corrections. Its geometric expression is an envelope, not a guaranteed observed growth rate.
• Deterministic branch comparisons and stochastic counterfactuals need different specifications. Nonparametric adjustment-based observational identification requires causal assumptions and action coverage. Extrapolating unsupported action effects requires additional structural assumptions; a simulator score alone does not establish identification.
• Closed-loop model errors affect real states through chosen actions. Re-observation avoids direct prediction substitution but does not guarantee smooth action response or stability.
• Transfer must specify what changed. Dynamics, observations, occupancy, objectives, and constraints can affect usefulness in different ways.
• In the stated toy, winners across the five metrics are Aggressive, Delayed, Delayed, Delayed, and Conservative. Those rankings are conditional on the horizon, input sequence, initial state, and controller.
• A scalar deployment score can be meaningful under a declared objective. Use the diagnostic vector to expose assumptions and untested behavior that the scalar would otherwise leave implicit.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about the world-model evaluation ladder.