Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Autoregressive Token World Models › Exposure Bias and Long-Horizon Generation

Michael BrenndoerferJune 27, 20266 min read

World Models HandbookAll chapters

Part 5 of 12World-Model Architectures

Chapter 27Autoregressive Token World Models › Exposure Bias and Long-Horizon Generation

Reading section 6 of 11 · Exposure Bias and Long-Horizon Generation · 6 minSection 6 of 11 · 6 min

Section 6 of 11 · 6 min read

Autoregressive Token World Models › Exposure Bias and Long-Horizon GenerationOne section · Sections in this chapter6 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Pure teacher forcing uses recorded histories; autonomous rollout uses generated world tokens after its supplied initial context. The question is whether those histories differ in distribution and how the model behaves on them.

The MechanismLink Copied

During rollout, world-token contexts are generated conditionally on the initial observation and supplied actions; those supplied inputs are not themselves sampled from the model. Model errors can shift later contexts and provoke further errors, but may also leave them in distribution or self-correct. Flawless-path probability is a different quantity. Let $S_t$ denote success at step $t$ and $p_t=\Pr(S_t\mid S_1,\ldots,S_{t-1})$ . Then the chain rule gives

\begin{aligned} \Pr[\text{flawless through } H] &= \prod_{t=1}^{H} p_t && \text{(conditional success probabilities)} \\ &= p^H && \text{(all } p_t \text{ equal } p\text{)} \end{aligned}

where:

$p_t$ : the probability of exact success at step $t$ conditional on all earlier successes
$H$ : the rollout horizon
$p$ : a constant conditional success probability along surviving prefixes; independent equal-accuracy steps are one sufficient special case

At $p = 0.99$ and $H = 100$ , that is about 37%. At $p = 0.95$ , under 1%.

The next cell plots the illustrative constant-success law $p^H$ across 400 steps. Later we estimate empirical flawless survival directly; we do not assume the measured errors are independent.

In[16]:

Code

import numpy as np

horizon_axis = np.arange(1, 401)
P_VALUES = [0.999, 0.99, 0.95, 0.90]
survival_curves = {p: p**horizon_axis for p in P_VALUES}

Out[17]:

Visualization

Log-scale chart of flawless-rollout probability decaying with horizon for four p values. — Flawless survival when every step has conditional success probability p given all earlier successes. Independence is sufficient but not required. At H=100, p=0.99 gives about 37% and p=0.95 gives under 1%.

Recovery can improve current-step accuracy but cannot restore a rollout's flawless-so-far status after a failure. Dependence can raise or lower survival relative to a product of marginal accuracies. For example, perfectly shared success events give survival $p$ at every horizon, whereas independent successes give $p^H$ . Pairwise correlation alone does not determine higher-order intersections. A uniform conditional failure probability at least $\delta>0$ on every surviving prefix implies survival at most $(1-\delta)^H$ ; merely nonzero marginal errors do not establish that bound.

Exposure Bias

Here exposure bias means a mismatch between recorded-history and generated-history distributions. Such mismatch can worsen future predictions, but neither mismatch nor monotonically increasing error is inevitable. Flawless survival measures accumulated exact-match failures, not exposure bias by itself.

Visual artifacts require diagnosis. Biased transitions can drift, depending on propagation gains; inconsistent patch predictions can impair coherence. Blur may arise from tokenizer reconstruction or an averaging decoder, but categorical sampling selects token IDs rather than averaging their decoded values. A token-only rollout feeds those IDs back, not necessarily a blurred decoded image. An object may disappear through representation or prediction errors even while its evidence remains in context; eviction matters only when no retained information preserves the needed distinction.

It is important to separate two error sources that produce similar-looking drift, because they have different cures:

Model error and feedback: clean one-step tests can reveal prediction errors, while rollout tests additionally expose their composition under feedback. Properly labeled generated-context training may help, but improvement and its cause require controls.
Environmental uncertainty: a correct stochastic model reproduces a conditional distribution, not the particular realization in a recorded trajectory. Independent valid draws can disagree even with no distribution mismatch. Compare generated samples with repeated environment outcomes or a validated conditional reference; dispersion of model samples alone cannot distinguish correct uncertainty from a diverse wrong model.

MitigationsLink Copied

The mitigation toolbox connects sequence modeling and control. Each intervention has its own target, information requirements, and possible side effects; no general ranking follows from the list.

Scheduled sampling (Bengio et al., 2015) replaces some recorded prior inputs with model outputs during training. It helped particular reported tasks, but changes ordinary data likelihood. Huszár's analysis shows that its perturbed objective can be statistically inconsistent. Treat its schedule and outcomes as empirical choices rather than a universally principled repair.
DAgger and generated-context training. DAgger aggregates expert-labeled states visited by a learner and reduces imitation learning to no-regret online learning. Its guarantees concern a suitable policy iterate under that reduction's assumptions, not monotone improvement of arbitrary neural rollouts. It needs an appropriate labeling oracle, not necessarily a simulator. Our later procedure is only inspired by this idea: it performs online corrective fine-tuning without dataset aggregation and uses reference-trajectory targets, so the DAgger theorem does not transfer.
Input perturbations and output regularization target different mechanisms. Context perturbation trains on altered inputs; label smoothing regularizes output probabilities. Numerical token-ID distance has no inherent semantic meaning. Robustness to perturbations in a defined observation or representation metric need not extend to large changes or unstable multi-step propagation.
Multi-step losses supervise predictions farther into a rollout. Continuous intermediates can support pathwise gradients; hard categorical choices require a specified estimator, relaxation, or surrogate. Shorter horizons may ease gradient estimation, while longer horizons cost more and can expose different errors. Neither horizon guarantees useful gradients or a valid likelihood interpretation; the exact objective matters.
Decoding choices such as temperature, top- $k$ , or nucleus truncation change diversity and mode preference. Low model probability does not mean an event is wrong, and a confident argmax can be wrong. Conservative decoding may suppress errors or valid rare outcomes; repetition can occur in some regimes. Measure these effects on the intended distribution rather than assuming coherence improves.
Fresh observations can reduce purely open-loop drift when the environment remains available. Re-reading generated history can preserve earlier information but supplies no independent truth correction. Replanning with observations does not eliminate model bias or confer robustness without sensing, estimation, and control assumptions.

Evaluate interventions against the properties they target. Paired image distances, distributional video metrics, exact-match rates, and controller returns answer different questions. A single prespecified metric can be useful evidence about one property, but is not a complete assessment of a world model.