Planning, Control, and Policy Evaluation › Limitations and Impact

Everything in this chapter is measured inside a low-dimensional, deterministic, fully observed toy. The system has two state variables, one scalar action, no measurement noise, no partial observability, no discrete events, and no perception stack. That is enough to make the evaluation framework concrete and to demonstrate the planning-horizon dilemma, the predicted-versus-actual gap, the ranking and selection protocols, and the transfer protocol on a system you can inspect line-by-line. It is not enough to say anything about how these effects play out in a 64×64 grayscale Atari environment, a video-based robot manipulation task, or a large-scale interactive game engine. The magnitudes of exploitation, ranking inversion, and transfer gap all depend on the difficulty of the model fit, the dimension of the state and action spaces, and the structure of the reward.

The specific learned model in this chapter is deliberately the simplest possible: linear in $(s, a)$ with a bias. Replacing it with a neural model, an ensemble, or a diffusion head can change the numerical results, but an equivalent predictive map or unchanged selected actions can preserve them. The evaluation question remains: what does the composed model-planner-policy-environment system earn, on which held-out cases, at which interaction budget, and against which baselines? The architecture of $\hat f$ enters as one component in a system, not as an end in itself. Publications that report only predictive accuracy and image fidelity are reporting on a rung of the ladder, not on the system.

Finite-set regret does not bound global suboptimality, and high unweighted pairwise agreement can hide a costly reversal between particular policies. The true-dynamics controller is a same-family heuristic reference, not a local upper bound. Reported standard deviations describe the 20 realized episode returns conditional on the disclosed seeds. They omit variation across training datasets and repeated planner streams at each initial state. Agarwal et al. (2021) explains why independent-run variation and appropriate interval estimates matter in RL evaluation.

The computational accounting in this chapter is a proxy, not a wall-clock measurement. Vectorized NumPy over $K$ candidates is not the same cost as the same number of candidate evaluations in a PyTorch model with a recurrent core or a transformer over a token sequence. Wherever throughput matters, wall-clock latency and memory footprint must be measured on the target hardware. The $K \cdot H$ proxy is useful for relative comparisons of planning algorithms at matched search size, but not for latency claims.

Transfer here changes one coefficient of deterministic nonlinear dynamics; the learned surrogates are affine. It does not test a linear-Gaussian environment family, visual shift, partial observability, or structural mechanism changes. A positive few-shot score alone would not establish either adaptation benefit or source-pretraining benefit; frozen and matched-budget target-only baselines address different comparisons. Our observed adaptation comparison is essentially a null result.

The MPC configurations share random shooting. Learned-model variants use affine surrogates, while the true-dynamics reference uses the exact nonlinear transition. PD is a separate model-free feedback comparator. We do not compare random shooting with CEM, gradient-based shooting, or learned policy priors. Substituting those components could change the results and would require its own resource accounting and evaluation.

The final limitation is the one that motivates the next chapter. Every return, regret, agreement, and transfer number in this chapter is conditional on the specific initial-state distributions, the specific evaluation horizons, the specific candidate sets, the specific source pretraining data, the specific target adaptation data, and the specific random seeds we happened to draw. Flip any of these and the ranking can change. Comparisons between systems are only meaningful when these conditions are matched or explicitly accounted for. Constructing datasets, benchmark suites, splits, and statistical summaries that make such comparisons reproducible across labs is the subject of Chapter 65, Datasets, Benchmarks, and Experimental Design. That chapter is not a restatement of what we did here; it is the discipline that makes what we did here comparable to what everyone else will do.