Failure Modes and Model Exploitation › Planner Exploitation and Adversarial Trajectories

Random shooting can propose and evaluate candidates uniformly in an action box while selecting them nonuniformly by predicted score. With multiple iid candidates and a fixed finite score that varies under the proposal law, minimum-score selection favors lower scores and changes that law. With one candidate, or constant scores and first-index tie-breaking, the selected law is unchanged. This concerns selected plans, not uniform candidate proposal or evaluation. An optimistic selected prediction need not mean poor absolute task performance, but it warrants comparing prediction and execution under the same conventions.

For costs, define selected optimism as $J_{\rm ref}-\hat J$ after minimizing $\hat J$; for returns, use $\hat R-R_{\rm ref}$ after maximizing $\hat R$. The relevant question is conditional error among selected candidates. Zero correlation between a score and its error does not imply independence or zero selected mean error. For example, let $Q$ be uniform on $\{-1,0,1\}$ and $E=Q^2-2/3$. Both $\mathbb E[E]$ and $\operatorname{Cov}(Q,E)$ are zero, yet selecting the largest of four independent $Q$ draws gives expected selected $E=4/27$. A sufficient zero-mean condition is $\mathbb E[E_i\mid Q_1,\ldots,Q_N]=0$ for every candidate, with selection based on those scores. The toy below does not establish such a condition or a general monotonic relation between correlation and optimism.

A large prediction error, or a large return gap between predicted and executed, is not by itself proof of exploitation. Several other things can produce a gap:

• Reward timing. If the reference emits rewards one step later than the training data assumed, even a perfect dynamics model can look wrong.
• Discounting. A different discount factor or terminal condition can change the return without model exploitation.
• Terminal conditions. Early termination can change return relative to a surrogate that runs to horizon, depending on omitted rewards and the terminal-payoff convention.
• Random-stream differences. If the reference and the surrogate share a random seed but not the same noise realization, the gap can be an artifact.
• Implementation errors. A sign flip, an off-by-one in the buffer, or a shape mismatch can produce gaps that look like exploitation but are bugs.

A protocol mismatch can be plan-dependent and budget-sensitive too. With exact dynamics, consider reward sequences $(1,0)$ and $(0,4)$. A scorer using discount $0.5$ ranks them as $1$ and $2$, while reference undiscounted returns are $1$ and $4$. Enlarging a nested pool from the first plan to both changes the selected reference-minus-predicted return gap from $0$ to $2$. That sign is pessimism under the return-optimism convention, but the example suffices to refute budget invariance. Check timing, discount, termination, and reward implementations before attributing a changing gap to model exploitation.

The laboratory controls these alternatives by sharing the cost definition, initial states, candidate pools, and termination rules, and by rescoring candidates with exact reference dynamics. A gap that changes with search under these controls supports selection-sensitive scoring error in this toy. Exploitation need not be off-support, need not worsen monotonically with budget, and need not produce a nonzero slope if the same optimistic candidate wins every pool.

Let's operationalize this. We will run the same planner on two different scoring models: the surrogate and the reference (which we can use as a "scoring oracle" because we control the reference process in simulation). Both use the same action pool, the same horizon, and the same objective. The only difference is which dynamics they roll out with. And then we cross-evaluate: every plan selected by either scorers is executed against the reference.

For each initial state, draw one pool from the fixed box $[-5,5]^4$ and reuse nested prefixes at increasing budgets. Retaining earlier candidates and keeping their scores fixed makes the minimum predicted cost nonincreasing. The proposal support does not widen with budget; more draws only provide more opportunities within that same box. Executed cost and optimism need not be monotone.

The unit of replication is a case: one initial state from $\mathcal U([-0.05,0.05])^2$ and one candidate pool of $2048$ plans from $\mathcal U([-5,5])^4$. The $60$ cases are paired across scorers and nested budgets; the baseline executes $(1,1,1,1)$ on each case. Pairing controls these inputs rather than removing all case variation. The cost plot reports separate means and case-level standard errors, not errors across candidate draws. The optimism plot first forms each case's executed-minus-predicted difference, then reports its mean and standard error. For any paired outcomes, $\operatorname{Var}(Y_1-Y_0)=\operatorname{Var}(Y_1)+\operatorname{Var}(Y_0)-2\operatorname{Cov}(Y_1,Y_0)$. Common additive effects cancel, but heterogeneous effects remain; pairing reduces variance relative to independent outcomes when the corresponding covariance is positive, not unconditionally.

Now the results.

Three patterns are visible in the figures and worth stating, together with what is not guaranteed.

With nested pools and unchanged scores, adding candidates cannot increase the minimum predicted cost. That limited guarantee does not cover arbitrary planners, noisy rescoring, or replaced pools, and it says nothing monotonic about executed cost.

Every budget uses the same $[-5,5]^4$ action box. More candidates can reveal combinations with overly favorable surrogate scores, but do not widen support or necessarily increase chosen command magnitude. Off-range commands are not all optimistic: $[-4,4,0,0]$ has predicted cost $13$ and reference cost $5.6875$, giving cost optimism $-7.3125$. The issue is selection of favorable errors, not membership outside the action interval alone.

The displayed optimism is positive at every tested budget and larger at high budgets than at the smallest, but it peaks and then declines. This particular search/scorer combination exposes optimism; it does not establish that every increase in search effort makes exploitation worse.

Exact deterministic reference scoring gives equal predicted and executed cost for each plan under the same initial state and conventions, hence zero per-plan optimism in this control. It selects the best plan in its sampled pool, not necessarily the global optimum. A zero gap does not require identical selected action sequences across all implementations or equally scored baselines, nor does statistical calibration guarantee equality for every stochastic realization.

Compare the selected plan with a declared no-search baseline on the tested objective. Worse cost is a regression on that metric; equal cost gives no measured improvement on it. Neither comparison by itself diagnoses exploitation or determines total deployment value, which may include other costs and constraints.

Ordinary optimization can produce adversarial-looking trajectories without an external attacker by selecting favorable scoring errors. The actuator example has saturated commands, not a modeled collision or damage process, so it does not establish that selected plans are physically unsafe to a human observer. Selection can amplify the consequences of existing errors without increasing each local state-prediction error.

An external adversary and ordinary score optimization differ in who chooses inputs and with what intent. Their failure mechanisms and defenses can overlap: constraints and score validation may address both. Part XII Chapter 3 is the planned security context; this example alone does not establish a disjoint class of mitigations.

A receding-horizon controller uses a new observation, state estimate, or belief to replan after executing part of a plan. It need not have direct access to the true state, and closed-loop control is not synonymous with repeated search. Replanning can change visited states and commitments enough to avoid or mitigate an exploitation pattern. It does not by itself correct a fixed scoring defect, and exploitation need not recur at every replanning call. Our laboratory executes open-loop sequences, so it does not measure that feedback effect.