Planning, Control, and Policy Evaluation › The Evaluation Target: Model, Planner, Policy, Environment

Let $s_t \in \mathbb{R}^{n}$ be the environment state, $a_t \in \mathcal{A}$ the action, $o_t$ the observation, and $r_t$ the reward. The deterministic reference transition is $s_{t+1}=f(s_t,a_t)$; a surrogate rollout substitutes $\hat f$ for $f$. Rewards here use the current state and action, $r_t=r(s_t,a_t)$. A policy selects actions from available observations or states. A deterministic feedback policy can be written $a_t=\pi(s_t)$; random-shooting MPC also depends on its internal random stream, which we denote by $\omega$. The planner scores candidate sequences inside its model, while the executed policy applies the selected first action and replans. Keeping these roles distinct clarifies what each reported return measures.

Classical control distinguishes the plant, controller, and predictive model. In this notebook, identification minimizes an empirical next-state prediction loss, while control evaluation measures closed-loop return. Identification can also be designed around control-specific criteria; generic prediction loss is not its only possible objective. A planner may be embedded in a controller, but candidate scoring and executed feedback remain different operations.

With partial observability, a model-based belief update uses transition and observation likelihoods, and planning can operate on the resulting belief state. The belief-state formulation of POMDPs makes that distinction explicit. Setting $o_t=s_t$ removes sensing and belief-estimation errors from this toy. It also means the examples cannot test failures specific to partial observability. Those settings are covered in Part VII.

The ladder from Chapter 61 covers one-step prediction, fixed-action open-loop rollouts, counterfactual interventions, closed-loop decisions, and transfer. These are complementary tests, not nested certificates. A model can perform well on a prediction metric yet support poor decisions; a controller can compensate for some prediction errors.

Teacher-forced prediction uses reference conditioning states rather than recursively predicted states. A fixed-action open-loop rollout removes observation corrections, allowing prediction errors to propagate without guaranteeing monotonic growth. Intervention tests can alter actions, states, or mechanisms; a selected-state reset is only one design. Closed-loop decisions evaluate the trajectories generated by the controller's own actions and feedback. Transfer evaluates specified changed conditions. Passing an early proxy does not certify a later task outcome, and failing it does not necessarily imply task failure.

Chapters 62 and 63 develop perceptual and state-level diagnostics; Chapter 61 introduces the full ladder, and this chapter expands its decision-facing tests. A controller that succeeds on one task has not thereby demonstrated faithful dynamics or general transfer. We report three kinds of information:

• A return metric computed by rolling the closed-loop policy in the true environment on a fixed held-out set of initial states.
• A resource accounting of the reported real-interaction and candidate-transition counts. This notebook does not measure wall-clock latency or memory footprint.
• A stress condition (a horizon, a shifted environment, a candidate policy set) that makes the returned number meaningful rather than survivable only in a lucky regime.

Return describes the tested outcome; resource counters describe its cost; stress conditions describe the scope of the test. Together they support interpretable comparisons. A result on one shifted gain or horizon establishes performance there, not robustness to every admissible shift.

A return without resource counters cannot establish sample or compute efficiency. A resource count can be audited independently, but a return comparison without a specified initial-state distribution is under-specified. Paired cases can improve comparison precision; appropriately replicated unpaired comparisons can also be valid.

A model-based controller can underperform because its model misrepresents relevant dynamics or rewards, or because its search, horizon, objective, or policy family is inadequate even with exact dynamics. Task infeasibility is another possibility. These causes interact. We use the same random-shooting planner with true dynamics as a reference, not as a global optimum or upper bound.

For a local model-substitution diagnostic, hold initial cases, rewards, search settings, and candidate random streams fixed while replacing learned dynamics with true dynamics. If the true-dynamics controller also performs poorly relative to a feasible baseline, investigate search, horizon, objective, and policy-family limitations. If substitution improves return, it provides evidence that the model choice matters under that protocol, not a certificate identifying a unique cause. Feedback trajectories can diverge even with matched streams. More search might help or hurt either controller, so test that remedy rather than ruling it out.