Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Failure Modes and Model Exploitation › Compounding Rollout Error

Michael BrenndoerferAugust 6, 202610 min read

World Models HandbookAll chapters

Part 12 of 12Reliable World Models

Chapter 67Failure Modes and Model Exploitation › Compounding Rollout Error

Reading section 3 of 10 · Compounding Rollout Error · 10 minSection 3 of 10 · 10 min

Section 3 of 10 · 10 min read

Failure Modes and Model Exploitation › Compounding Rollout ErrorOne section · Sections in this chapter3 / 10 ↓

Sections in this chapter10

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Begin with prediction propagation under matched actions. It can contribute to a planning failure, but its presence or prevalence is not implied by the existence of other mechanisms.

Run the reference and surrogate from the same initial state using the same realized action sequence $\{a_0,\ldots,a_{T-1}\}$ . Define $e_t=\|\hat{s}_t-s_t\|$ in one fixed state norm. A discrepancy at one step changes the input to the next, so small one-step errors can accumulate or amplify. They need not do so: contraction, cancellation, and exact predictions can keep the realized discrepancy small. We will bound it under explicit domain and action conditions.

Let's derive the bound. Write $\hat{f}_\theta$ for the surrogate and $f$ for the reference. Define the surrogate rollout and the reference rollout in parallel:

\hat{s}_{t+1} = \hat{f}_\theta(\hat{s}_t, a_t), \qquad s_{t+1} = f(s_t, a_t).

Now subtract and insert a zero:

\hat{s}_{t+1} - s_{t+1} = \bigl[\hat{f}_\theta(\hat{s}_t, a_t) - f(\hat{s}_t, a_t)\bigr] + \bigl[f(\hat{s}_t, a_t) - f(s_t, a_t)\bigr].

Take norms and apply the triangle inequality:

e_{t+1} \le \bigl\|\hat{f}_\theta(\hat{s}_t, a_t) - f(\hat{s}_t, a_t)\bigr\| + \bigl\|f(\hat{s}_t, a_t) - f(s_t, a_t)\bigr\|.

The first term measures same-input approximation error at the surrogate's current state. The second evaluates the reference map at two different states. A uniform state-Lipschitz bound is a property of that fixed reference map, domain, action set, and norm; the value of this term depends on the states reached and therefore on the surrogate too. The surrogate states must be included in the domain, not just the reference states. Assume:

(A1) In the fixed norm, $\varepsilon\ge0$ bounds $\|\hat{f}_\theta(s,a)-f(s,a)\|$ for every $s\in\mathcal D$ and every considered action. Both trajectories stay in $\mathcal D$ throughout the comparison.
(A2) In the same norm, $L\ge0$ bounds reference state sensitivity uniformly over the same actions: $\|f(s,a)-f(s',a)\|\le L\|s-s'\|$ for all $s,s'\in\mathcal D$ and all considered $a$ .

then the recursion collapses to a familiar scalar form:

\boxed{\;e_{t+1} \le L e_t + \varepsilon.\;}

The recurrence bounds the next discrepancy by a propagated-error budget $Le_t$ plus an approximation budget $\varepsilon$ . These are upper bounds, not prescribed increments in the actual error. Directions, signs, and how tightly $L$ and $\varepsilon$ fit the visited states determine how much of that budget is realized.

Unrolling from $e_0$ :

e_t \le L^t e_0 + \varepsilon \sum_{j=0}^{t-1} L^j = L^t e_0 + \varepsilon \cdot \frac{L^t - 1}{L - 1} \quad (L \ne 1),

and for $L = 1$ the sum stays linear:

e_t \le e_0 + \varepsilon t.

The upper envelope has three regimes. These describe the bound, not three mandatory shapes for the actual error:

$0\le L<1$ : the envelope tends to $\varepsilon/(1-L)$ as its initial-error term vanishes. The actual discrepancy need not approach that plateau. This bounds state error under (A1)–(A2); it is not a task-safety guarantee. A smaller valid $\varepsilon$ lowers the envelope, though the observed error need not change proportionally.
$L=1$ : the envelope is $e_0+\varepsilon t$ . A scalar update $s'=s+u$ has unit state sensitivity for a fixed action. A double integrator is different: in a normalized state $(x,v)$ with $A=\left[\begin{smallmatrix}1&0.5\\0&1\end{smallmatrix}\right]$ , its Euclidean operator norm is about $1.280776$ , not $1$ . Its unit eigenvalues do not give a unit operator-norm bound: $A^t$ contains $0.5t$ . Constant velocity bias can produce quadratic position discrepancy. Halving $\varepsilon$ halves the additive part of this envelope, not necessarily the realized error or a nonzero initial-error term.
$L>1$ : the envelope contains a geometric factor. To judge a horizon, compare the full bound, including $\varepsilon$ and $e_0$ , with a relevant state-error tolerance. For $L=1.05$ , $\varepsilon=0.001$ , and $e_0=0$ , the bound first exceeds $1$ at step $81$ , not after a generic handful of steps. An exact surrogate with matched initial state and actions has zero discrepancy at every horizon, even when the reference is expansive.

Three caveats matter for interpreting that formula.

First, the bound is not an equality. Propagated contributions can cancel, and loose constants or directions with low sensitivity can also leave the realized error well below it. For a scalar recurrence $d_{t+1}=\lambda d_t+b_t$ , the contribution of $b_k$ at time $t$ is $\lambda^{t-1-k}b_k$ . Cancellation concerns these weighted contributions, not simply the sum of raw bias signs. Alternating biases need not remain bounded when $\lambda>1$ ; the examples below illustrate particular traces rather than a general cancellation guarantee.

Second, it is not a theorem that realized error increases monotonically. You can have a trajectory where the reference and the surrogate happen to drift back together for a few steps before separating again. The bound describes the envelope, not the path. Reporting a single number for "rollout error at horizon $T$ " collapses a whole error trace into one value and hides this shape.

Third, both trajectories must remain inside the domain where (A1)–(A2) hold. In a scalar coefficient-mismatch example, $\hat f(s)-f(s)=\delta s$ with $\delta\ne0$ , no finite uniform approximation bound exists over the whole real line. For a vector mismatch $\Delta A s$ , growth depends on direction and can vanish on the mismatch's nullspace. An additive bias instead admits a uniform bound on its norm. Leaving a validated domain invalidates its local bound unless it is extended or re-established.

Let's make it concrete with a scalar example. Take

s_{t+1} = \lambda s_t, \qquad \hat{s}_{t+1} = \lambda \hat{s}_t + \varepsilon,

Here $\varepsilon\ge0$ is the constant additive bias and $L=|\lambda|$ . There is no action in this scalar illustration; adding the same input term $Ba_t$ to both maps would cancel in their difference, not be absorbed into $\lambda$ . With $d_0=0$ and $\lambda\ge0$ , the constant-bias trace attains the geometric-sum envelope. Negative $\lambda$ or changing bias signs can make it strict. The approximation error is state-independent, so this example separates propagation from changing local approximation quality.

In[6]:

Code

def signed_error_trace(lam, eps, T, alt_sign=False, e0=0.0):
    """Signed propagation of an additive bias eps through a scalar linear map."""
    e = e0
    trace = [e]
    for t in range(T):
        bias = (eps if (t % 2 == 0) else -eps) if alt_sign else eps
        e = lam * e + bias
        trace.append(e)
    return np.array(trace)


def geometric_bound(lam, eps, T, e0=0.0):
    """Evaluate the nonnegative geometric envelope without a near-unit branch.

    This is floating-point evaluation, not directed-rounding certification.
    """
    if (
        isinstance(T, (bool, np.bool_))
        or not isinstance(T, (int, np.integer))
        or T < 0
    ):
        raise ValueError("T must be a nonnegative integer")
    if not np.all(np.isfinite([lam, eps, e0])) or eps < 0:
        raise ValueError(
            "lam and e0 must be finite; eps must be finite and nonnegative"
        )
    L = abs(float(lam))
    bounds = np.empty(T + 1, dtype=float)
    bounds[0] = abs(float(e0))
    with np.errstate(over="ignore"):
        for t in range(T):
            bounds[t + 1] = L * bounds[t] + float(eps)
    return bounds


T_STEPS = 20
EPS_BIAS = 0.05

error_profiles = {
    lam: {
        "constant": signed_error_trace(lam, EPS_BIAS, T_STEPS),
        "alternating": signed_error_trace(
            lam, EPS_BIAS, T_STEPS, alt_sign=True
        ),
        "bound": geometric_bound(lam, EPS_BIAS, T_STEPS),
    }
    for lam in (0.9, 1.0, 1.05)
}

Now let's visualise the three regimes in separate figures, and add the alternating-sign case to show how loose the bound can be.

Out[7]:

Visualization

Constant-sign error matches a saturating bound; alternating-sign error is smaller after step one. — Contractive reference with lambda = 0.9: the constant-sign additive-bias trace equals the geometric bound at each step, while the alternating-sign trace is below it after step one. At step twenty the absolute errors are approximately 0.4392 and 0.0231, respectively, a magnitude ratio of nineteen; the asymptotic bound is 0.5.

Constant-sign error grows linearly; alternating-sign error alternates between zero and 0.05. — Reference with lambda = 1.0: constant-sign error and the upper bound grow linearly, while alternating-sign errors cancel every second step. The curves illustrate that equal per-step error magnitudes need not give equal recursive errors.

Constant-sign error grows geometrically; alternating-sign error never exceeds the same bound. — Expansive reference with lambda = 1.05: constant-sign error equals the geometric bound and exceeds 1.6 at step twenty. Alternating-sign error never exceeds the bound and is strictly lower after step one in this construction; alternation does not establish a general stability guarantee.

The contractive panel illustrates both tightness and cancellation. For $\lambda = 0.9$ and $\varepsilon = 0.05$ , the constant-sign trace equals the geometric bound at each step and approaches $\varepsilon / (1-\lambda)=0.5$ . At step twenty, the constant-sign signed trace is approximately $+0.4392$ and the alternating-sign signed trace is approximately $-0.0231$ . Their absolute errors, the quantities plotted, are approximately $0.4392$ and $0.0231$ , with a magnitude ratio of nineteen. Both traces start at zero and coincide at step one, so the reduction is not present at every step. Using the bound as a planning constraint may be conservative when errors cancel, but it is tight in the aligned-sign construction. The same uniform per-step error magnitude can therefore accompany different recursive errors. These deliberately chosen signed-error patterns illustrate a mechanism; they do not establish where every trained model falls between them.

These observations suggest several questions to ask when choosing a rollout horizon.

Horizon effects depend on sensitivity, bias structure, initial error, and the relevant task tolerance. The three plotted references differ in $\lambda$ ; they are not one fixed model with three behaviors. Shortening a horizon can lower the envelope even in a contractive system close to $L=1$ , and an expansive upper bound need not describe the realized error. Compare proposed horizons on the actual system instead of labeling one regime harmless and another inevitably unusable.
One-step error magnitude is not a complete rollout description; error directions and temporal structure also matter. A related strategy in Janner et al.'s MBPO paper (NeurIPS 2019) uses short model rollouts branched from real states and studies rollout length alongside model and policy-distribution error. That is not a theorem about the particular alternating-bias experiment here, nor a guarantee that any short rollout has small error.
Talvitie's self-correcting-model work (2017) trains on model-generated inputs paired with next states from a parallel reference trajectory. Its analysis uses restricted conditions including deterministic reference dynamics and blind rollout policies, with shared action sequences. This hallucinated-training target is not generally the same as $\|\hat f(z,a)-f(z,a)\|$ at one common input, so improving it does not automatically lower (A1)'s uniform $\varepsilon$ . It does not change a fixed reference map's uniform $L$ on a fixed domain and norm; those statements should not be mistaken for this bound proving the method's guarantees.
Matched actions, rather than how those actions were generated, are the key comparison. The derivation remains pathwise valid for a shared feedback-generated action sequence if (A1)–(A2) hold along both paths. If each path chooses its own action, additionally bound the action difference: with reference action sensitivity $L_a$ and state sensitivity $L_s$ , $e_{t+1}\le L_s e_t+L_a\|\hat a_t-a_t\|+\varepsilon$ on the required domains. Feedback may damp or amplify this mismatch. Neither inequality alone proves closed-loop stability or task safety.

Compounding can occur while evaluating candidate rollouts as well as while executing a chosen plan. Next we distinguish a wrong task objective from prediction bias, then ask what selection does to attractive scoring errors.