Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Failure Modes and Model Exploitation › Notation and the distinctions that must survive

Michael BrenndoerferAugust 6, 20266 min read

World Models HandbookAll chapters

Part 12 of 12Reliable World Models

Chapter 67Failure Modes and Model Exploitation › Notation and the distinctions that must survive

Reading section 2 of 10 · Notation and the distinctions that must survive · 6 minSection 2 of 10 · 6 min

Section 2 of 10 · 6 min read

Failure Modes and Model Exploitation › Notation and the distinctions that must surviveOne section · Sections in this chapter2 / 10 ↓

Sections in this chapter10

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Before we touch a single failure, it is worth fixing the vocabulary. The whole chapter depends on keeping these objects separate.

Objects and their notation

$s_t \in \mathcal{S}$ : the reference state, the physical or otherwise "true" variable that generates the data.
$o_t \in \mathcal{O}$ : the observation, which can omit information in $s_t$ . Our dynamical examples expose the state directly; the later cue example deliberately does not. Part III: Representing Agents and Worlds develops this distinction.
$z_t$ : a learned latent state. A latent coordinate is not automatically a sufficient statistic for $s_t$ .
$h_t$ : hidden recurrent memory carried by a model across time. It may contain information that no single $z_t$ does.
$a_t \in \mathcal{A}$ : the action the agent issues.
$r_t$ : reference reward; $c_t$ : reference cost. When comparing the cost and return conventions, we use $r_t=-c_t$ . A learned reward or cost head is distinct from the intended task objective.
$f$ or $p$ : reference dynamics; $\hat f_\theta$ : surrogate dynamics, which may be estimated from data or explicitly constructed as in these examples.
$\pi$ : a policy specifying an action or an action distribution from the information available to the controller, such as a state, observation history, or belief. A planner selects actions or action sequences by searching with a model.

Keep these objects separate when interpreting a failure. A lossy observation or latent representation can hide a needed distinction. A planner can preferentially select a model's optimistic errors. A supplied objective can differ from the intended task even with exact dynamics. These are different questions, not mutually exclusive labels: the opening already combines dynamics bias, selection, and a transition-support violation.

Two distinctions are especially useful when deciding what to test.

First, prediction quality is not decision quality. A fixed lower average supervised loss is better on that loss and evaluation distribution, but need not improve decisions under a different selected distribution or objective. A model with larger average error can still support better control if its decision-relevant predictions are better or its use is more conservative. Separate the predictions from how the agent uses them. Part VII: Planning and Agency develops the planning side; here we examine their interaction.

Second, model bias and objective misspecification live in different objects. Bias is error in surrogate dynamics or a predicted reward; misspecification is a supplied optimization target that differs from the intended task. Exact dynamics can optimize a wrong target, and an aligned target can still be evaluated through biased dynamics. Editing a target does not make a fixed transition map accurate, nor does refitting transitions make a fixed target aligned. Behavioral remedies can nevertheless overlap: a constraint or penalty can discourage a harmful plan while leaving its dynamics prediction wrong. Distinguish repairing the defective object from mitigating its consequences.

Let's write down the reference process and the surrogate so we can point at them.

In[3]:

Code

import itertools

import numpy as np

## --- The reference process: a 1-D point mass with a saturated actuator ---
DT = 0.5  # seconds per step
U_MAX = 1.0  # hardware saturation bound on the commanded thrust
X_GOAL = 1.0  # target position at every step of the horizon
H = 4  # planning and execution horizon (in steps)
A_PLAN = 5.0  # planner's candidate action range, deliberately wider than U_MAX


def reference_step(s, a):
    """Reference dynamics: the actuator saturates, everything else is linear."""
    x, v = s
    a_eff = float(np.clip(a, -U_MAX, U_MAX))
    return np.array([x + v * DT, v + a_eff * DT])


def surrogate_step(s, a):
    """Constructed linear surrogate: exact for in-range actions, unclipped outside."""
    x, v = s
    return np.array([x + v * DT, v + a * DT])


def rollout(step_fn, s0, actions):
    """Roll out a fixed action sequence under the given dynamics."""
    s = np.asarray(s0, dtype=float)
    traj = np.empty((len(actions) + 1, 2))
    traj[0] = s
    for i, a in enumerate(actions):
        s = step_fn(s, a)
        traj[i + 1] = s
    return traj


def trajectory_cost(traj):
    """Sum of squared position error to X_GOAL over the horizon."""
    return float(np.sum((traj[1:, 0] - X_GOAL) ** 2))


def palette_colors(palette):
    """Return the shared style palette as a plain list of colors."""
    return (
        list(palette.values()) if hasattr(palette, "values") else list(palette)
    )


s0_default = np.array([0.0, 0.0])

surrogate_step and reference_step agree exactly for every state whenever $|a|\le1$ . They are constructed maps, not the result of a reported fitting run. A dataset restricted to that interval cannot distinguish these two transition rules from observations alone. Known actuator constraints or an appropriate model class can distinguish them without collecting out-of-range data. Their discrepancy is an unconstrained extension beyond the stipulated action support.

Compare three hand-chosen plans before running the separate randomized search experiment.

In[4]:

Code

## A plan that looks perfect to the surrogate, and one the hardware can actually run.
plan_shortcut = np.array([4.0, -4.0, 0.0, 0.0])
plan_feasible = np.array([1.0, 1.0, -1.0, -1.0])
plan_naive = np.ones(H)

surrogate_traj = rollout(surrogate_step, s0_default, plan_shortcut)
reference_traj = rollout(reference_step, s0_default, plan_shortcut)
feasible_traj = rollout(reference_step, s0_default, plan_feasible)
naive_traj = rollout(reference_step, s0_default, plan_naive)

predicted_cost = trajectory_cost(surrogate_traj)
executed_cost = trajectory_cost(reference_traj)
feasible_cost = trajectory_cost(feasible_traj)
naive_cost = trajectory_cost(naive_traj)

Out[5]:

Console

Shortcut plan [4.0, -4.0, 0.0, 0.0]
  surrogate predicted cost :   1.0000
  reference executed cost  :   2.6875
  optimism (exec - pred)   :   1.6875

Feasible plan [1.0, 1.0, -1.0, -1.0]
  reference executed cost  :   1.6250

Naive full-thrust plan [1.0, 1.0, 1.0, 1.0]
  reference executed cost  :   1.8750

The constructed shortcut predicts cost $1.0$ but executes at $2.6875$ , worse than the feasible comparator's $1.625$ and the all-positive comparator's $1.875$ . Calling an attractive scoring error model exploitation emphasizes selection of that error, not merely its existence. Restricting this toy's commands to $[-1,1]$ makes surrogate and reference scores agree for every considered plan. A finite random search can still miss the feasible comparator; no-search behavior depends on the chosen baseline. Neither restriction alone guarantees one of the displayed hand-chosen plans.

The examples that follow isolate complementary mechanisms. They are not a nested hierarchy: a wrong objective can occur with exact dynamics, while bias, propagation, and exploitation can occur together. The final workflow uses their distinctions to design comparisons, not to assign every failure exactly one label.