Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Failure Modes and Model Exploitation › Model Bias and Objective Misspecification

Michael BrenndoerferAugust 6, 20267 min read

World Models HandbookAll chapters

Part 12 of 12Reliable World Models

Chapter 67Failure Modes and Model Exploitation › Model Bias and Objective Misspecification

Reading section 4 of 10 · Model Bias and Objective Misspecification · 7 minSection 4 of 10 · 7 min

Section 4 of 10 · 7 min read

Failure Modes and Model Exploitation › Model Bias and Objective MisspecificationOne section · Sections in this chapter4 / 10 ↓

Sections in this chapter10

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

A model-based agent can fail because its predictions are wrong, because its supplied objective is wrong, or because both interact. Bad trajectories and low task performance do not by themselves distinguish these possibilities. The next controlled example keeps dynamics exact and changes only the objective.

Model bias vs. objective misspecification

Model bias is discrepancy in surrogate dynamics or a surrogate reward head. From rest in the opening toy, commanding $a=4$ predicts next velocity $2$ under the unclipped surrogate, while reference clipping gives $0.5$ . The actuator caps the command, not the vehicle's velocity at all future times. Better identification, known actuator structure, or constraints on queried actions can address this discrepancy or its consequences.

Objective misspecification is discrepancy between the supplied optimization target and the intended task. If the task requires stopping and the cost omits terminal velocity, accurate dynamics do not add that preference. Add an appropriate velocity term when it is a preference, or impose a stopping constraint when it is a hard requirement. A finite penalty and a hard constraint are not interchangeable.

The two failure axes are logically distinct but can coexist and interact. Changing a fixed dynamics map does not rewrite a fixed objective, and changing a fixed objective does not correct the map. Either intervention can still change which plan is selected and mitigate observed behavior. The following example isolates objective mismatch with exact dynamics rather than proving repairs never overlap.

The following counterexample keeps the dynamics exactly correct and changes only the objective. There is no transition-prediction error to remove: improving transition accuracy alone cannot supply an omitted stopping preference while the objective, candidate set, and planning protocol remain fixed. Other model-side interventions can change selected behavior; this example does not rule them out. Consider a double integrator with exact dynamics

x_{t+1} = x_t + v_t, \qquad v_{t+1} = v_t + a_t,

starting from $(x_0, v_0) = (0, 0)$ . The action is bounded by $|a_t| \le 1$ . Over four steps, a plan $(a_0, a_1, a_2, a_3)$ produces

x_4 = 3a_0 + 2a_1 + a_2, \qquad v_4 = a_0 + a_1 + a_2 + a_3.

With continuous $|a_i|\le1$ and four unit-time steps from rest, the reachable terminal position lies in $[-6,6]$ . Position alone does not specify stopping: $[1,1,-1,-1]$ reaches $x_4=4$ with $v_4=0$ . Indeed, under the rest condition $\sum_i a_i=0$ , $x_4=1.5a_0+0.5a_1-0.5a_2-1.5a_3\le4$ , with equality only at that plan. The proxy below penalizes terminal position error plus action energy but omits terminal velocity. Reaching the same position with nonzero velocity is therefore not ruled out.

J_{\text{proxy}}(a) = (x_4 - 4)^2 + 10^{-3} \sum_t a_t^2, \qquad J_{\text{intended}}(a) = (x_4 - 4)^2 + v_4^2 + 10^{-3} \sum_t a_t^2.

Both objectives are soft costs. The velocity term makes moving arrival more expensive; it does not reject every nonzero $v_4$ . We will enumerate a grid of five action values per step, giving $5^4=625$ plans, and compare its two minimizing plans. On this grid the intended-cost optimum stops. That need not be the continuous optimum: $[1,1,-1,-0.999]$ reaches $x_4=4$ with $v_4=0.001$ and intended cost $0.003999001$ , below the stopping plan's $0.004$ . Require $v_4=0$ explicitly if exact stopping is part of feasibility.

In[8]:

Code

A_GRID = np.array([-1.0, -0.5, 0.0, 0.5, 1.0])
PLAN_LEN = 4
X_TARGET = 4.0
ACTION_PENALTY = 1e-3

plans = np.array(list(itertools.product(A_GRID, repeat=PLAN_LEN)))

final_x = 3.0 * plans[:, 0] + 2.0 * plans[:, 1] + plans[:, 2]
final_v = plans.sum(axis=1)
action_penalty = ACTION_PENALTY * np.sum(plans**2, axis=1)

proxy_cost = (final_x - X_TARGET) ** 2 + action_penalty
intended_cost = (final_x - X_TARGET) ** 2 + final_v**2 + action_penalty

idx_proxy = int(np.argmin(proxy_cost))
idx_intended = int(np.argmin(intended_cost))

plan_proxy = plans[idx_proxy]
plan_intended = plans[idx_intended]

Out[9]:

Console

Number of enumerated plans: 625

Proxy-optimal plan    [1.0, 0.5, 0.0, 0.0]
  proxy cost          :   0.001250
  intended cost       :   2.251250
  final x, final v    :  4.000,  1.500

Intended-optimal plan [1.0, 1.0, -1.0, -1.0]
  proxy cost          :   0.004000
  intended cost       :   0.004000
  final x, final v    :  4.000,  0.000

The proxy's grid optimum, $[1,0.5,0,0]$ , reaches $x_4=4$ with $v_4=1.5$ : proxy cost $0.00125$ , intended cost $2.25125$ . The intended cost's grid optimum, $[1,1,-1,-1]$ , stops at the target with cost $0.004$ . This is an objective mismatch under exact dynamics. Moving arrival is penalized by the stated task; no collision or crash process is modeled. The calculation establishes the finite-grid contrast, not exact stopping for every continuous optimum.

The two objectives disagree not about position but about the velocity at arrival, and the selected plans make that tension visible.

In[10]:

Code

selection_x = final_x.copy()
selection_v = final_v.copy()
selection_proxy_index = idx_proxy
selection_intended_index = idx_intended

Out[11]:

Visualization

Scatter of final position versus velocity for 625 grid plans, highlighting the proxy grid optimum at the target but moving and the intended-cost grid optimum at the target at rest. — Final position and velocity of the 625 enumerated grid plans under exact dynamics. The proxy grid optimum reaches the target with velocity 1.5, while the intended-cost grid optimum arrives at rest. These soft objectives select different plans on this grid; the continuous optimum need not stop exactly.

The opening actuator failure and this objective failure isolate different defective objects. Adding terminal velocity to a cost does not make an unclipped map obey clipping; adding clipping does not express a missing task preference. A penalty or constraint can nevertheless steer selection away from a harmful candidate while the map remains wrong. Diagnose whether the intervention repairs predictions, aligns the target, or merely changes the chosen behavior.

A further distinction is reward-channel integrity. Everitt et al.'s analysis of reward tampering concerns inappropriate influence on the reward function or its inputs. Optimizing a fixed misspecified proxy need not involve such influence; tampering and misspecification can also coexist. Reward-channel protections, incentive design, objective specification, and model-scoring checks are complementary, not disjoint professional domains. Pan et al.'s proxy-reward study is relevant to optimization against an imperfect proxy, not evidence that every proxy/true-performance divergence is tampering.

The defects can coexist: a controller could combine an unclipped actuator model with a position-only proxy for a task that also requires stopping. This is a constructed possibility, not evidence of deployment prevalence. Separate questions lead to separate fixes:

Does the transition or reward head predict the reference quantities accurately on the inputs used for the decision? Test the relevant map and inputs, rather than inferring dynamics bias from a return gap alone.
Does the supplied objective express the intended task when evaluated on reference trajectories? Check missing terms, constraints, timing, discounts, and termination conventions.
Does selection preferentially favor favorable scoring errors? This can amplify consequences of an existing model defect; it is not an alternative that proves the model itself is sound.

Test prediction accuracy, task alignment, and selection effects under controlled comparisons. More than one can require attention. Changing one component at a time helps interpret a contrast, but downstream behavior and interactions mean a successful repair does not uniquely identify every cause.