Failure Modes and Model Exploitation › Failure Modes and Model Exploitation

Imagine a vehicle moving along a one-dimensional track, with position $x$, velocity $v$, and thrust command $a$. The actuator clips commands to $[-1,1]$. Suppose your training data contain only commands in that interval. A linear transition rule and the clipped reference rule agree exactly there, so those data alone do not identify their different out-of-range behavior. We will construct these two rules directly; the chapter does not claim to have collected a dataset or fitted a model.

Give the surrogate to an open-loop random-shooting planner: sample $N$ four-action sequences from a box such as $[-5,5]^4$, predict each rollout, and select the sequence with the lowest predicted cost. The stipulated task charges squared position error from $x=1$ at each of the four steps. The experiments execute the whole selected sequence. Applying only its first action and searching again after an observation would instead make this a receding-horizon controller; we discuss that possibility later but do not execute it here.

One constructed candidate is $[4,-4,0,0]$: thrust, reverse, then coast. The surrogate predicts cost $1.0$ and a terminal position of $1$. Executing the same commands in the clipped reference process gives terminal position $0.25$ and cost $2.6875$. These are squared-position costs in the toy's normalized units, not currency. This candidate illustrates an optimistic score; it is not asserted to be the output of a particular random search or of a fitted confidence model.

The story links three mechanisms. The surrogate is exact on the stipulated action interval but biased outside it. Selection can favor candidates whose predicted costs are erroneously low. The resulting transition can also violate the reference's action-conditioned possibilities. These mechanisms overlap: exploitation does not exonerate the model that supplied its attractive error. The position cost is the task specified here, not a misspecified proxy; a separate example will isolate objective misspecification. No external attacker or coding accident is needed for this constructed failure.

This is the first chapter of Part XII: Reliable World Models. Part XI: Evaluation and Understanding covered prediction, perceptual similarity, physical consistency, and planning performance. Here we ask why strong measurements on tested cases need not imply good decisions on other cases. We will develop distinct, sometimes overlapping failure mechanisms, reproduce selected CPU-sized examples, and use controlled comparisons to diagnose failures without promising a unique cause from every symptom.

An average validation loss weights inputs according to its evaluation distribution. It does not by itself bound error on every decision-relevant input. Score-based selection can change the candidate distribution, and allowed search can expose optimistic model errors that validation did not cover. For iid proposals, selecting the minimum of a fixed finite predicted score reweights their law when there is more than one candidate and the score varies under that law. With one candidate, or a constant score and first-index tie-breaking, the selected law is unchanged. Other failures below need no off-distribution search: the objective may be wrong, an observation may omit a cue, or a conditional mean may violate transition support. Good average prediction error can therefore coexist with poor decisions. Their possible severity also depends on task-cost and horizon bounds; the finite toy does not establish arbitrarily large loss.