Planning, Control, and Policy Evaluation › Model Exploitation and Policy Ranking

Model exploitation occurs when action selection favors trajectories whose predicted value is spuriously high because of model errors. A planner can prefer such trajectories without knowing that its predictions are wrong. This is a possible failure mechanism, not an inevitable property of every optimizer. Our fitted model has region-dependent prediction errors, but the measured comparisons below show pessimistic return estimates and no PD ranking reversal. They therefore do not demonstrate optimistic exploitation in this toy. They show how to measure return-prediction gaps and policy-selection fidelity without assuming that a failure must appear.

Selection can create optimism even when individual estimates are unbiased. For fixed candidate values and integrable, conditionally unbiased score errors, the expected selected score overstates the selected candidate's true value by a nonnegative amount; it can be zero. In an equal-value example with independent identically distributed scale-family noise, expected maximum error grows with candidate count and noise scale. Correlated errors, unequal true values, and candidate composition can change that behavior. These conditions distinguish the optimizer's curse from an unconditional claim that more search worsens control.

Several mechanisms can contribute to a return-prediction discrepancy:

• Dynamics error: $\hat f(s, a) \ne f(s, a)$ in the region the planner visits.
• Reward-model error: $\hat{r}(s, a) \ne r(s, a)$ even if the dynamics are perfect.
• Objective misspecification: the reward the planner optimizes is not the reward the environment's evaluation uses.

These are possible mechanisms, not an exact additive decomposition. Horizon truncation, changed feedback trajectories, planner randomness, and finite evaluation samples also affect what is compared. Different remedies address different mechanisms: improve relevant dynamics data or model class, check reward predictions, or align objectives. The toy uses the same specified reward everywhere and pairs planner streams, but this does not uniquely decompose the remaining discrepancy.

At a fixed position, $-(p^2+v^2)$ discourages speed while $-(p^2-v^2)$ rewards it. Accurate dynamics alone do not align those opposite objective preferences. In our experiments, $\hat r=r$; the MPC lookahead horizon still differs from the 30-step evaluation horizon.

Possible mitigations have different assumptions. PETS (Chua et al., 2018) propagates uncertainty from probabilistic ensembles through trajectory sampling; it is not an explicit ensemble-disagreement reward penalty. MOPO (Yu et al., 2020) uses uncertainty-penalized rewards in an offline model-based learning setting. Ensemble averaging can reduce unshared estimation variation, but shared errors can remain. State-conditioned action or state-action support constraints can restrict unsupported queries; marginal action bounds alone do not guarantee supported states. Shorter rollouts, terminal value estimates, and feedback replanning are other options, each requiring evaluation in its own setting.

Bootstrap resampling is one way to diversify ensemble fits, as in PETS. A terminal value estimate approximates reward beyond the rollout horizon instead of explicitly simulating every later step. Relative to executing one fixed open-loop plan, replanning performs a new search at each observation and therefore adds planning calls.

A penalty can sacrifice useful actions if uncertainty is miscalibrated or its scale is poorly chosen. An ensemble can agree on an incorrect prediction. A support constraint can exclude a useful unseen action. A shortened horizon can miss delayed reward, and a terminal value model can inherit training blind spots. Fresh observations let replanning react to discrepancies but do not guarantee correction of model error. These are mitigations to test, not safety certificates.

For the return-prediction comparison, run the same controller implementation in learned and true dynamics from identical initial cases. Each case reconstructs its planner stream with seed $110+i$ before both runs. Model-closed-loop return uses predicted states as controller inputs and as reward arguments; true-closed-loop return uses reference states. This is not the planner's local $H$-step candidate score. Even the controller that plans with true dynamics is evaluated inside the learned outer simulator for its model-predicted point below.

All three return estimates are pessimistic in the paired run. PD predicts about $-3.704$ and earns $-3.588$, a gap of $-0.116$. Learned-model MPC predicts $-4.932$ and earns $-4.612$, a gap of $-0.319$. True-dynamics MPC predicts $-5.080$ in the learned outer simulator and earns $-4.731$ in the reference environment, the largest negative gap here, about $-0.349$. These are model-closed-loop discrepancies, not evidence of optimistic exploitation or a unique causal decomposition.

The scatterplot compares true-environment and model-closed-loop mean return. Above the equality diagonal would mean optimism; all three points are below it. Streams are reconstructed before each paired run, while feedback trajectories and therefore chosen actions may still diverge between the two outer simulators.

Every point is below the equality diagonal. True-dynamics MPC has the largest negative discrepancy in this paired evaluation. Reporting only these model-predicted scores would understate the observed returns, so the useful lesson is to measure the discrepancy rather than assume optimism.

The diagonal is exact agreement between the two mean-return estimates. Departure measures return-prediction discrepancy; three policy points are not a full distributional calibration study.

The fixed candidate set tests scoring and ordering without adaptively generating candidates from model scores. Selecting the highest predicted score can still create selection optimism; fixing the set does not remove every selection effect. Good ordering on this finite set supports good selection within it, not on unseen policies. Poor ordering can compromise selection, but it does not prove that adding candidates or changing search cannot improve actual return.

Pairwise agreement is the fraction of pairs $(i,j)$ with $y_i\ne y_j$ for which $\mathrm{sign}(p_i-p_j)=\mathrm{sign}(y_i-y_j)$, where $p_i$ is the predicted mean and $y_i$ the measured reference mean. A predicted tie on non-tied truth counts as disagreement. If no true-score pair is eligible, the fraction is undefined and the code reports NaN. Selection regret is $\max_i y_i-y_{\hat i}$, with $\hat i=\arg\max_i p_i$ and first-index tie breaking. It is nonnegative for this finite set, not a gap to the globally optimal policy. Equality of finite-sample means is a numerical tie convention, not proof of equal population returns.

For these six zipped PD gain pairs, all 15 eligible pairs are concordant, agreement is 1.000, and both score lists select $k_p=8,k_d=5$. Finite-set regret is 0.000. This run demonstrates the diagnostic without producing a ranking failure. It does not establish order preservation outside this candidate set or the fixed evaluation cases.

The fixed candidate set can now be plotted directly, with the true best and model-selected points marked.

The scatter plot below places each PD candidate at its true return on the horizontal axis and its model-predicted return on the vertical axis. A star marks the best true candidate and an X the model-selected candidate. The two marks coincide in this run because the model selects the true best candidate. Their horizontal separation would measure finite-set selection regret if the selected candidates differed; here that regret is zero.