The World-Model Evaluation Ladder › Closed-loop planning

Closed-loop planning evaluates the model as part of a decision system. At each step the agent observes the real state, queries the model to choose an action, executes that action in the true environment, and observes again. For a deterministic environment the resulting state is $s_{t+1}=F(s_t,a_t)$; with stochastic dynamics it is sampled from $T(\cdot\mid s_t,a_t)$.

Re-observation prevents a predicted state from directly replacing the real next state. It does not sever the effect of model error on the trajectory: predictions influence the action, and the action influences future real states. Feedback can correct some errors, but an unstable or poorly chosen controller can still fail.

Small prediction perturbations need not cause small action changes. Consider actions $-1$ and $+1$, true next state $F(0,a)=a$, and target zero. An exact predictor gives equal squared costs, so an optimizer with a tie rule can choose $-1$. Perturb only the predicted $+1$ successor from $1$ to $1-\epsilon$, with $0<\epsilon<1$. The prediction changes by only $\epsilon$, but the unique preferred action is now $+1$, changing the real successor by two. Smooth action sensitivity or stable correction requires additional conditions on the optimizer, objective, constraints, and closed-loop dynamics.

Let $J(\pi)$ be expected cumulative cost in the true environment, with a common objective, initial-state distribution, noise law, horizon, and policy class $\Pi$. If $\pi_\theta\in\Pi$, define

$$G=J(\pi_\theta)-\inf_{\pi\in\Pi}J(\pi)\geq0.$$

Assume the costs and infimum are finite. If an optimal policy $\pi_{\mathrm{opt}}$ attains that infimum, the gap is $J(\pi_\theta)-J(\pi_{\mathrm{opt}})$. For rewards being maximized, use $\sup_{\pi\in\Pi}J_{\mathrm{reward}}(\pi) -J_{\mathrm{reward}}(\pi_\theta)$ instead. The sign depends on whether the objective is a cost or a reward.

Model approximation, finite-data estimation, planner suboptimality, and distribution mismatch are diagnostic categories, not universally independent additive quantities. An exact decomposition would need defined intermediate models and policies. We use the categories to investigate the observed gap.

A planner can exploit model errors by selecting actions whose predicted cost is favorable but whose real cost is not. This is one possible failure mechanism of optimizing through an inaccurate model. Greater search capacity does not invariably worsen performance: it can also find better actions. The direction depends on the errors, objective, constraints, and search procedure.

Three practical points follow.

• Feedback and rollout test different uses: replanning from observations can compensate for recursive model drift. Good tracking in that setting does not certify a long open-loop simulation.
• Report the whole decision system: the result belongs to the model, planner, objective, constraints, observation convention, and compute budget together. Changing any of them can change the ranking.
• Separate objective validity from decision optimization: a model can predict transitions accurately while the controller optimizes the wrong objective for deployment. State the intended objective and the scored objective before attributing a decision failure to dynamics.

The worked example uses observed states and a one-step grid search. It does not test belief-state inference, long-horizon planning, or safety.