Planning, Control, and Policy Evaluation

Suppose a learned-model planner predicts a return of $-3.2$ but earns $-48$ when its controller runs in the reference environment. These numbers are hypothetical, not outputs from the notebook below. The contrast matters because a predicted rollout and an executed closed loop answer different questions: the first is about the surrogate dynamics, while the second is about the model, planner, policy, and environment acting together.

A gap can arise from model error, but it can also reveal mismatched evaluation protocols or an implementation bug. It is not inevitable: a model that is exact on the visited trajectories, with matching rewards and controller random streams, can predict return exactly. Lower average one-step loss can nevertheless coexist with worse closed-loop return. Prediction loss averages over its evaluation distribution; a planner may query a different distribution and amplify errors that matter to action selection. Good planning does not require this shift, and unsuccessful planning can shift too. Measure both predicted and executed return before attributing their difference.

This chapter develops that comparison and the resource accounting around it. Chapter 62, Perceptual and Generative Evaluation, covers image-quality and distributional diagnostics. Chapter 63, State, Physics, Causality, and Memory Evaluation, covers state readouts, physical constraints, interventions, and memory diagnostics. Chapter 61 introduced the full evaluation ladder, including closed-loop decisions and transfer. Here we expand the decision-facing tests: what return does a composed controller earn on specified tasks, distributions, and budgets?

We examine four questions. Task return and sample efficiency connect achieved performance to real-interaction budgets. Planning horizon and compute tradeoffs compare search depth and breadth. Model exploitation and policy ranking test selection-sensitive prediction errors and the ordering of fixed candidate policies. Zero-shot, few-shot, and cross-domain transfer specify what stays frozen, what adapts, and which deployment conditions change.

These questions share a composed-system framework, but they are not marginals of one fixed trajectory distribution. Changing a planner, policy, or environment can change the distribution itself; functionally equivalent controllers or changes outside visited states need not do so. Ranking compares scores across policies; it need not analyze distribution tails. Transfer can change just one environment parameter, as it does below. The diagnostics can disagree: a biased return estimate can preserve policy order, and a costly planner can still achieve good return. Joint reporting makes those distinctions visible.

A concrete lesson runs through all four faces. The evaluation target is a composed system, not a component. When we say "the new model is better," we must always be able to say "better at what return, on what initial-state distribution, at what compute and interaction budget, and compared against which baselines on which held-out cases." Without that, the numbers are decoration.

Model quality has several separable dimensions. Predictive fidelity compares dynamics and reward predictions with reference outcomes under a stated distribution. Representation quality asks what task-relevant information a state representation makes accessible. Uncertainty quality includes calibration or coverage of specified predictive quantities, sharpness, and useful discrimination between uncertain and well-supported predictions. Decision usefulness asks what a controller achieves in execution. All four can use environment-derived reference data; the decision test additionally executes actions in a closed loop. Strong predictive or uncertainty scores alone do not certify control performance.

This separates two questions: is the model accurate under the chosen predictive metric, and is it useful for this decision problem? Small distribution-averaged errors can still matter greatly if they affect reward-sensitive states or action ordering. Conversely, errors in task-irrelevant quantities may have little effect on a particular controller. Name the metric and task rather than treating accuracy and usefulness as synonyms.

Let $s_t \in \mathbb{R}^{n}$ be the environment state, $a_t \in \mathcal{A}$ the action, $o_t$ the observation, and $r_t$ the reward. The deterministic reference transition is $s_{t+1}=f(s_t,a_t)$; a surrogate rollout substitutes $\hat f$ for $f$. Rewards here use the current state and action, $r_t=r(s_t,a_t)$. A policy selects actions from available observations or states. A deterministic feedback policy can be written $a_t=\pi(s_t)$; random-shooting MPC also depends on its internal random stream, which we denote by $\omega$. The planner scores candidate sequences inside its model, while the executed policy applies the selected first action and replans. Keeping these roles distinct clarifies what each reported return measures.

Classical control distinguishes the plant, controller, and predictive model. In this notebook, identification minimizes an empirical next-state prediction loss, while control evaluation measures closed-loop return. Identification can also be designed around control-specific criteria; generic prediction loss is not its only possible objective. A planner may be embedded in a controller, but candidate scoring and executed feedback remain different operations.

With partial observability, a model-based belief update uses transition and observation likelihoods, and planning can operate on the resulting belief state. The belief-state formulation of POMDPs makes that distinction explicit. Setting $o_t=s_t$ removes sensing and belief-estimation errors from this toy. It also means the examples cannot test failures specific to partial observability. Those settings are covered in Part VII.

The ladder from Chapter 61 covers one-step prediction, fixed-action open-loop rollouts, counterfactual interventions, closed-loop decisions, and transfer. These are complementary tests, not nested certificates. A model can perform well on a prediction metric yet support poor decisions; a controller can compensate for some prediction errors.

Teacher-forced prediction uses reference conditioning states rather than recursively predicted states. A fixed-action open-loop rollout removes observation corrections, allowing prediction errors to propagate without guaranteeing monotonic growth. Intervention tests can alter actions, states, or mechanisms; a selected-state reset is only one design. Closed-loop decisions evaluate the trajectories generated by the controller's own actions and feedback. Transfer evaluates specified changed conditions. Passing an early proxy does not certify a later task outcome, and failing it does not necessarily imply task failure.

Chapters 62 and 63 develop perceptual and state-level diagnostics; Chapter 61 introduces the full ladder, and this chapter expands its decision-facing tests. A controller that succeeds on one task has not thereby demonstrated faithful dynamics or general transfer. We report three kinds of information:

• A return metric computed by rolling the closed-loop policy in the true environment on a fixed held-out set of initial states.
• A resource accounting of the reported real-interaction and candidate-transition counts. This notebook does not measure wall-clock latency or memory footprint.
• A stress condition (a horizon, a shifted environment, a candidate policy set) that makes the returned number meaningful rather than survivable only in a lucky regime.

Return describes the tested outcome; resource counters describe its cost; stress conditions describe the scope of the test. Together they support interpretable comparisons. A result on one shifted gain or horizon establishes performance there, not robustness to every admissible shift.

A return without resource counters cannot establish sample or compute efficiency. A resource count can be audited independently, but a return comparison without a specified initial-state distribution is under-specified. Paired cases can improve comparison precision; appropriately replicated unpaired comparisons can also be valid.

A model-based controller can underperform because its model misrepresents relevant dynamics or rewards, or because its search, horizon, objective, or policy family is inadequate even with exact dynamics. Task infeasibility is another possibility. These causes interact. We use the same random-shooting planner with true dynamics as a reference, not as a global optimum or upper bound.

For a local model-substitution diagnostic, hold initial cases, rewards, search settings, and candidate random streams fixed while replacing learned dynamics with true dynamics. If the true-dynamics controller also performs poorly relative to a feasible baseline, investigate search, horizon, objective, and policy-family limitations. If substitution improves return, it provides evidence that the model choice matters under that protocol, not a certificate identifying a unique cause. Feedback trajectories can diverge even with matched streams. More search might help or hurt either controller, so test that remedy rather than ruling it out.

Task return is the sum of rewards earned by a policy in an episode of finite length $T$, $G(\pi, s_0) = \sum_{t=0}^{T-1} \gamma^{t}\, r(s_t, a_t)$ with $s_0 \sim \mu$, where:

• $G(\pi, s_0)$: the total discounted reward accumulated over the episode starting from initial state $s_0$
• $s_0 \sim \mu$: the initial state is drawn from the stated initial-state distribution $\mu$
• $T$: the finite episode horizon (number of steps)
• $\gamma \in (0, 1]$: the discount factor (here we set $\gamma = 1$, giving an undiscounted finite-horizon objective)
• $r(s_t, a_t)$: the scalar reward received at step $t$ in state $s_t$ under action $a_t$
• $a_t = \pi(s_t)$ for a deterministic policy: randomized MPC additionally depends on its planner stream $\omega$
• $s_{t+1} = f(s_t, a_t)$: the next state produced by the true environment transition dynamics

The discount factor and episode horizon belong to the task definition. Here $\gamma=1$ and $T=30$, so the code sums 30 unweighted rewards with no terminal reward. Changing discount or horizon can change policy rankings, but different reward timing does not force a ranking reversal. Comparisons should use the same objective, or explicitly present the different objectives as different tasks.

For a randomized controller, write the expected evaluation return as $J(\pi)=\mathbb{E}_{s_0\sim\mu,\,\omega}[G(\pi,s_0;\omega)]$, where $\omega$ contains the planner's random draws. A stochastic environment would add transition randomness to that expectation. The deterministic PD controller needs no $\omega$; random-shooting MPC does. Our finite estimates use 20 held-out initial states and one prescribed stream per case, seeded $110+i$ for case $i$. Reconstructing those streams pairs comparisons. These estimates do not average over independent training runs or repeated planner streams at each state, and geometric state coverage alone does not establish estimation precision.

An expected-return claim must state its initial-state distribution $\mu$ and any other randomness being averaged. Other legitimate policy claims concern success probability, tail risk, safety, or latency and require their own defined quantities. Different initial-state distributions answer different return questions unless the comparison explicitly accounts for them.

Initial states influence the trajectories and model queries a policy generates. Starting near the origin does not guarantee that a controller stays there, nor does a farther starting state guarantee worse return. Measure visitation and return if those mechanisms are the claim. Here a fixed uniform sample on $[-1,1]^2$ defines the held-out cases for all source controllers and the paired target experiment.

Sample efficiency describes performance as a function of real-environment interaction budget, not a raw ratio $J/N$. That ratio is especially unhelpful for negative rewards, where its ordering can change simply by changing the reward offset. A learning curve evaluates systems at several disclosed interaction budgets, using either cumulative online learning or separate fits. Independent seeds and uncertainty estimates are needed to characterize variability; their number should be justified by the precision of the comparison. Track these interaction buckets separately:

• Source pretraining interactions: data used before the target adaptation experiment. Source and target can share an environment; the domains must be stated.
• Target adaptation interactions. Real trajectories collected in the deployment environment and used to update any parameter (model, policy, value, planner hyperparameter).
• Validation rollouts. Interactions used to select among hyperparameters or checkpoints. This is not free: those rollouts also consumed actions.
• Test rollouts. Interactions used to report the metric, with no update or selection step consuming their outcomes.

Separating the counters prevents two accounting errors: treating synthetic model transitions as newly collected environment data, and omitting interactions used for selection. Report each bucket so readers can compare the resources their deployment constrains.

Synthetic transitions generated by $\hat f$ are compute, not real samples. A planner that rolls the model $K \cdot H$ times per decision did not collect $K \cdot H$ real environment steps. It spent $K \cdot H$ model evaluations. Both quantities matter, but they belong to different budgets. Real interactions may constrain a deployment when collecting data is expensive or dangerous; model evaluations may constrain it when inference is slow or costly. A method can be sample-efficient and compute-inefficient, or the reverse. Report both budgets against the deployment's actual limits.

A horizon sweep can mix longer lookahead, fewer candidates, and longer model rollouts. A candidate-count sweep changes search adequacy at a fixed model. Neither alone establishes a data-efficiency advantage. To measure that advantage, vary the real-interaction budget and evaluate the resulting learner or checkpoint under a consistent test protocol.

The remaining experiments use a deterministic damped double integrator with two state variables and a known nonlinear transition. This keeps return and resource accounting inspectable. The same evaluation questions apply when studying ensemble methods, imagination-based policies, or value-equivalent models, but those methods need their own implementations and protocols; this notebook does not benchmark them.

The testbed is a deterministic point mass with position and velocity state, damped dynamics, and nonlinear quadratic drag. Its reference transition is a known formula, so we can substitute it into the same planner without claiming to solve control optimally. A neural surrogate would not remove that reference formula; an unknown physical plant would make exact-dynamics substitution unavailable.

The system is a 1D damped point mass with quadratic drag. The state is $s=(p,v)\in\mathbb{R}^2$, the action is a scalar bounded acceleration $a\in[-1,1]$, and its discrete source transition is $v_{t+1}=(1-d)v_t+\Delta t\,g\,a_t-c_{\mathrm{drag}}v_t|v_t|$, followed by $p_{t+1}=p_t+\Delta t\,v_{t+1}$. These are the toy's discrete update rules, not an exact continuous-time solution.

where:

• $v_t$: velocity at time $t$
• $p_t$: position at time $t$
• $a_t$: scalar bounded acceleration applied at time $t$
• $\Delta t$: integration timestep, $0.1$
• $d$: linear damping coefficient, $0.05$
• $c_{\mathrm{drag}}$: quadratic drag coefficient in the discrete velocity update, $0.1$; it is distinct from the return discount $\gamma$
• $g$: actuator gain, set to a source value of $1.0$ and (for transfer) a target value of $0.7$
• $|v_t|$: absolute velocity, capturing the sign-symmetric drag effect

The first line updates velocity from linear damping, the scaled control input, and quadratic drag. The second line updates position by semi-implicit (post-update velocity) Euler integration. The reward is a quadratic tracking-and-control penalty to the origin, $r(s, a) = -\bigl( p^2 + 0.05\, v^2 + 0.01\, a^2 \bigr)$, where:

• $p^2$: position penalty, driving the state toward the origin
• $0.05\, v^2$: velocity penalty, discouraging high speed
• $0.01\, a^2$: control effort penalty, discouraging large actions
• the overall negative sign: penalizes deviations rather than rewarding them, so higher return means staying closer to the origin with less effort

The quadratic term $c_{\mathrm{drag}}v|v|$ makes the true transition nonlinear, while the learned surrogate is affine in state and action. Least squares solves the empirical fitting problem within that class. This is a deliberate class mismatch, not an injected corruption of the fitted coefficients.

Empirical least-squares optimality does not rule out overfitting or finite-sample estimator variation. There is no transition or observation noise here, but the sampled training trajectories still affect the fitted coefficients. The affine class also cannot represent the quadratic drag everywhere. Held-out errors and control tests measure the consequences of that fit; they do not decompose all error into a single bias term.

Even for a simple PD controller whose behavior is not chosen by the model, the model's prediction of PD's return differs from reality. In this run the model predicts a lower return, so its error is pessimistic rather than optimistic. Replacing PD with a model-driven planner can change the gap, but neither its sign nor its cause follows from the PD comparison alone.

This is fixed-budget accounting, not a sample-efficiency learning curve. The model fit uses 3000 source transitions. The fixed PD gains require no fitting transitions, but evaluating PD costs 20 episodes of 30 steps, or 600 environment transitions. Compare both performance and disclosed costs; do not call these 600 transitions 600 rollouts or divide the negative return by a sample count.

A receding-horizon planner solves a local finite-horizon optimization at each decision step. Given the current state $s_t$, it evaluates candidate action sequences of length $H$, picks the one that maximizes predicted cumulative reward, executes only the first action, advances one step in the environment, and repeats. The policy emitted by this loop is not a static map; its behavior depends on where inside the model the search lands at each step.

Receding-horizon control executes the first action of a plan and refreshes the current state before planning again. This can limit the effect of trajectory prediction errors, but the chosen first action still depends on the candidate reward scores. Here each score uses $H$ pre-transition reward terms, evaluated at the current state and $H-1$ recursively predicted states; the final computed successor has no terminal reward and is not scored. Replanning is not a guarantee that those forecasts or actions are sound.

Random shooting draws $K$ candidate sequences, each containing $H$ actions, scores their model rollouts, and executes the first action of the highest-scoring candidate. This implementation performs $K\cdot H$ candidate transition evaluations per decision, excluding reward evaluation and ancillary overhead. It is a useful matched-search-count proxy when comparing these configurations, not a complete hardware-independent compute or latency metric.

Random shooting is a simple baseline. Cross-entropy methods iteratively refit a sampling distribution around elite candidates; gradient-based shooting uses differentiable dynamics. Here the random-shooting policy is determined by its state input, model, settings, and evolving random stream. Fresh policy factories reconstruct that stream per evaluation case so comparisons can share draws.

Latency depends on implementation, hardware, batch size, memory layout, and model cost. Timing claims can generalize only with supporting measurements or a justified performance model. Matching $K\cdot H$ answers a specific depth-versus-breadth question; latency-matched comparisons and resource-performance frontiers answer other legitimate questions.

Under a fixed $K\cdot H$ budget, increasing $H$ exposes later rewards while reducing candidate count. Longer sequences explore a larger action space and use more recursively predicted states. Prediction errors can propagate, but their magnitude need not increase monotonically. Only the chosen first action is executed before replanning; all $H$ candidate reward stages contribute to the score.

These effects need not balance at the same horizon across tasks or search algorithms. Longer lookahead can help delayed rewards, while finite random shooting can struggle to cover longer action sequences. A horizon sweep measures the resulting behavior, not a unique decomposition of the causes.

Talvitie (2017) analyzes composed prediction errors and self-correcting models, rather than prescribing universally short MPC horizons. MBPO (Janner et al., 2019) uses short model rollouts branched from real data for policy learning; that is distinct from this MPC horizon sweep. Wang et al. (2019) explicitly studies a planning-horizon dilemma in model-based RL. These papers motivate measuring horizon sensitivity, not a theorem that shorter always wins.

The planner below owns its seeded random generator. Each evaluated episode creates a fresh policy with seed $110+i$. Learned and true-dynamics planners with the same $(H,K)$ therefore use the same candidate arrays at corresponding decision indices. Different $(H,K)$ settings reshape draws into different candidate sequences, so the horizon sweep is not an identical-action-sequence intervention.

Do not generalize this sweep into a rule that shorter horizons are better. Delayed-reward tasks may require deeper lookahead or a terminal value estimate. Nor is the true-dynamics heuristic an upper bound: at several horizons the learned-model heuristic earns a higher observed return. Search, horizon, and model-quality sweeps under declared budgets can test possible remedies; this figure alone does not select one.

Equal candidate-transition counts need not produce equal latency. This NumPy implementation batches over $K$ and loops over $H$, while a neural implementation has its own batching and sequence costs. Measure wall-clock latency on the relevant hardware if latency is the claim; do not infer that the two configurations must differ on every possible machine.

Model exploitation occurs when action selection favors trajectories whose predicted value is spuriously high because of model errors. A planner can prefer such trajectories without knowing that its predictions are wrong. This is a possible failure mechanism, not an inevitable property of every optimizer. Our fitted model has region-dependent prediction errors, but the measured comparisons below show pessimistic return estimates and no PD ranking reversal. They therefore do not demonstrate optimistic exploitation in this toy. They show how to measure return-prediction gaps and policy-selection fidelity without assuming that a failure must appear.

Selection can create optimism even when individual estimates are unbiased. For fixed candidate values and integrable, conditionally unbiased score errors, the expected selected score overstates the selected candidate's true value by a nonnegative amount; it can be zero. In an equal-value example with independent identically distributed scale-family noise, expected maximum error grows with candidate count and noise scale. Correlated errors, unequal true values, and candidate composition can change that behavior. These conditions distinguish the optimizer's curse from an unconditional claim that more search worsens control.

Several mechanisms can contribute to a return-prediction discrepancy:

• Dynamics error: $\hat f(s, a) \ne f(s, a)$ in the region the planner visits.
• Reward-model error: $\hat{r}(s, a) \ne r(s, a)$ even if the dynamics are perfect.
• Objective misspecification: the reward the planner optimizes is not the reward the environment's evaluation uses.

These are possible mechanisms, not an exact additive decomposition. Horizon truncation, changed feedback trajectories, planner randomness, and finite evaluation samples also affect what is compared. Different remedies address different mechanisms: improve relevant dynamics data or model class, check reward predictions, or align objectives. The toy uses the same specified reward everywhere and pairs planner streams, but this does not uniquely decompose the remaining discrepancy.

At a fixed position, $-(p^2+v^2)$ discourages speed while $-(p^2-v^2)$ rewards it. Accurate dynamics alone do not align those opposite objective preferences. In our experiments, $\hat r=r$; the MPC lookahead horizon still differs from the 30-step evaluation horizon.

Possible mitigations have different assumptions. PETS (Chua et al., 2018) propagates uncertainty from probabilistic ensembles through trajectory sampling; it is not an explicit ensemble-disagreement reward penalty. MOPO (Yu et al., 2020) uses uncertainty-penalized rewards in an offline model-based learning setting. Ensemble averaging can reduce unshared estimation variation, but shared errors can remain. State-conditioned action or state-action support constraints can restrict unsupported queries; marginal action bounds alone do not guarantee supported states. Shorter rollouts, terminal value estimates, and feedback replanning are other options, each requiring evaluation in its own setting.

Bootstrap resampling is one way to diversify ensemble fits, as in PETS. A terminal value estimate approximates reward beyond the rollout horizon instead of explicitly simulating every later step. Relative to executing one fixed open-loop plan, replanning performs a new search at each observation and therefore adds planning calls.

A penalty can sacrifice useful actions if uncertainty is miscalibrated or its scale is poorly chosen. An ensemble can agree on an incorrect prediction. A support constraint can exclude a useful unseen action. A shortened horizon can miss delayed reward, and a terminal value model can inherit training blind spots. Fresh observations let replanning react to discrepancies but do not guarantee correction of model error. These are mitigations to test, not safety certificates.

For the return-prediction comparison, run the same controller implementation in learned and true dynamics from identical initial cases. Each case reconstructs its planner stream with seed $110+i$ before both runs. Model-closed-loop return uses predicted states as controller inputs and as reward arguments; true-closed-loop return uses reference states. This is not the planner's local $H$-step candidate score. Even the controller that plans with true dynamics is evaluated inside the learned outer simulator for its model-predicted point below.

Transfer evaluation has a protocol problem before it has a metric problem. The words "zero-shot," "few-shot," and "cross-domain" are used loosely, so let's fix operational definitions that survive peer review.

Transfer needs a protocol before a metric because the same word, "zero-shot," can mean at least three different things. It can mean the model is frozen and no target data is seen at all. It can mean the model is frozen but a small amount of target data was used to select hyperparameters. It can mean the model is frozen but the planner's hyperparameters were tuned on target rollouts. These settings use different information and can have different interaction budgets and expected performance. Calling all of them zero-shot conceals those protocol differences. The operational definitions below make the information access and budgets explicit.

• Zero-shot means no target-domain training, adaptation, or selection information is used before evaluation. Model, policy, planner, and learned hyperparameters are frozen. Target rollouts used for checkpoint or hyperparameter selection count as validation interactions and violate this strict zero-shot definition, even without gradient updates.
• Few-shot permits a stated, bounded number of target-environment transitions (call it $N_{\text{few}}$) and a stated number of adaptation steps (gradient updates, closed-form refits, or meta-updates). Both numbers are reported. Define the target adaptation budget as $N_{\text{few}}$ (number of target-environment transitions allowed) and $K_{\text{adapt}}$ (number of adaptation steps). The target evaluation rollouts are separate from the $N_{\text{few}}$ transitions used for adaptation; if an evaluation rollout is used to select among adaptation settings, it is not an evaluation rollout anymore.
• Cross-domain transfer names the specific shift: shifted dynamics (different $f$), shifted observations (different sensor model), shifted reward (different task objective), or shifted task (different target). A claim like "our method transfers well" is not operational without the shift label.

To test a source-pretraining benefit at a given target-data budget, compare against a target-only baseline with the same allowed target interactions. Matching that budget removes a data-quantity confound, not every source of variation. More target data do not guarantee better performance.

The source and target environments differ only in actuator gain, from $g=1.0$ to $g=0.7$; observations, reward, and task stay fixed. We also pair initial cases and planner streams for the frozen-controller comparison. The physical source actuator coefficient exceeds the target coefficient by $1/0.7-1\approx43\%$ relative to the target. This concerns the magnitude of a nonzero action's incremental actuator effect, not every total next-state prediction. The fitted affine coefficient need not equal the physical coefficient exactly.

Everything in this chapter is measured inside a low-dimensional, deterministic, fully observed toy. The system has two state variables, one scalar action, no measurement noise, no partial observability, no discrete events, and no perception stack. That is enough to make the evaluation framework concrete and to demonstrate the planning-horizon dilemma, the predicted-versus-actual gap, the ranking and selection protocols, and the transfer protocol on a system you can inspect line-by-line. It is not enough to say anything about how these effects play out in a 64×64 grayscale Atari environment, a video-based robot manipulation task, or a large-scale interactive game engine. The magnitudes of exploitation, ranking inversion, and transfer gap all depend on the difficulty of the model fit, the dimension of the state and action spaces, and the structure of the reward.

The specific learned model in this chapter is deliberately the simplest possible: linear in $(s, a)$ with a bias. Replacing it with a neural model, an ensemble, or a diffusion head can change the numerical results, but an equivalent predictive map or unchanged selected actions can preserve them. The evaluation question remains: what does the composed model-planner-policy-environment system earn, on which held-out cases, at which interaction budget, and against which baselines? The architecture of $\hat f$ enters as one component in a system, not as an end in itself. Publications that report only predictive accuracy and image fidelity are reporting on a rung of the ladder, not on the system.

Finite-set regret does not bound global suboptimality, and high unweighted pairwise agreement can hide a costly reversal between particular policies. The true-dynamics controller is a same-family heuristic reference, not a local upper bound. Reported standard deviations describe the 20 realized episode returns conditional on the disclosed seeds. They omit variation across training datasets and repeated planner streams at each initial state. Agarwal et al. (2021) explains why independent-run variation and appropriate interval estimates matter in RL evaluation.

The computational accounting in this chapter is a proxy, not a wall-clock measurement. Vectorized NumPy over $K$ candidates is not the same cost as the same number of candidate evaluations in a PyTorch model with a recurrent core or a transformer over a token sequence. Wherever throughput matters, wall-clock latency and memory footprint must be measured on the target hardware. The $K \cdot H$ proxy is useful for relative comparisons of planning algorithms at matched search size, but not for latency claims.

Transfer here changes one coefficient of deterministic nonlinear dynamics; the learned surrogates are affine. It does not test a linear-Gaussian environment family, visual shift, partial observability, or structural mechanism changes. A positive few-shot score alone would not establish either adaptation benefit or source-pretraining benefit; frozen and matched-budget target-only baselines address different comparisons. Our observed adaptation comparison is essentially a null result.

The MPC configurations share random shooting. Learned-model variants use affine surrogates, while the true-dynamics reference uses the exact nonlinear transition. PD is a separate model-free feedback comparator. We do not compare random shooting with CEM, gradient-based shooting, or learned policy priors. Substituting those components could change the results and would require its own resource accounting and evaluation.

The final limitation is the one that motivates the next chapter. Every return, regret, agreement, and transfer number in this chapter is conditional on the specific initial-state distributions, the specific evaluation horizons, the specific candidate sets, the specific source pretraining data, the specific target adaptation data, and the specific random seeds we happened to draw. Flip any of these and the ranking can change. Comparisons between systems are only meaningful when these conditions are matched or explicitly accounted for. Constructing datasets, benchmark suites, splits, and statistical summaries that make such comparisons reproducible across labs is the subject of Chapter 65, Datasets, Benchmarks, and Experimental Design. That chapter is not a restatement of what we did here; it is the discipline that makes what we did here comparable to what everyone else will do.

Decision evaluation concerns a composed model-planner-policy-environment system. Report its executed return under stated initial states, objectives, randomness, resource budgets, and baselines. Predictive quality alone does not certify that outcome, and one successful task does not certify faithful dynamics or general transfer.

Task return is the finite reward sum along an environment rollout. Expected return averages over initial states and, where relevant, planner and environment randomness. Sample efficiency describes achieved performance as a function of real-interaction budget, not return divided by sample count. This notebook is a fixed-budget experiment rather than a learning curve. Source training, adaptation, validation, and test transitions are distinct from synthetic model-search calls.

The horizon sweep matches $K\cdot H=640$ candidate transition evaluations per decision. Both learned and true-dynamics MPC peak at $H=5$ in this run, then decline; PD has the higher mean throughout. This is evidence about the specified search configuration, not a universal short-horizon rule or a unique model-error diagnosis. Candidate-transition counts are a proxy; latency and memory require separate measurement.

Model exploitation is a possible selection-sensitive failure in which model errors make chosen actions look spuriously valuable. Under conditionally unbiased candidate estimates, selection can create nonnegative expected optimism, with growth claims requiring additional assumptions about candidate values and errors. Our paired controller estimates are instead pessimistic, and the six-policy ranking test has agreement 1 and finite-set regret 0. These diagnostics measure discrepancies without inventing an exploitation or ranking failure. Uncertainty propagation, penalties, support constraints, shorter rollouts, terminal values, and replanning have tradeoffs, not safety guarantees.

Zero-shot and few-shot protocols must identify frozen components, target interactions, adaptation operations, and selection data. The target experiment pairs cases and candidate streams and discloses 400 adaptation transitions plus 2400 target evaluation transitions across four controllers. Frozen and both adapted MPC means are nearly identical on this run; no appreciable adaptation gain or source-pretraining benefit is demonstrated.

The experiments make the protocol inspectable: fixed cases and seeds, explicit return calculations, paired controller streams, and separate data and compute counters. They do not measure wall-clock latency, memory, confidence intervals, or population-wide transfer. Chapter 65 develops datasets, splits, benchmarks, and statistical summaries that make such comparisons reproducible beyond a single toy run.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about planning, control, and policy evaluation.