Video Generators as Candidate World Models › Prediction Horizons: Short, Long, and Mode Collapse
World Models HandbookAll chapters Video Generators as Candidate World Models › Prediction Horizons: Short, Long, and Mode CollapseOne section · Sections in this chapter5 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Longer horizons expose more opportunities for model error, although an exact model or a contracting system need not show increasing error at every step. A single average can hide where a rollout fails, so report performance by horizon and condition.
Short-horizon prediction (a handful of frames). Slow motion and stable backgrounds can make frame similarity an easy baseline. Check whether changing a feasible action changes the near future in the expected direction. Prompt sensitivity alone does not demonstrate response to a timed action.
Mid-horizon prediction (hundreds of milliseconds to seconds in a chosen task). A contact event can reveal a failure that frame-similarity scores obscure: bodies interpenetrate, or a grasp does not change the object's motion. The relevant time span depends on the task and frame rate, not on a universal model-failure threshold.
Long-horizon prediction (many transitions at the task's sampling rate). Recursive prediction can compound bias, stochastic error, or off-distribution feedback. Probe position, object identity and count, and camera pose separately: a model might maintain one while losing another.
In a recursive generator, sampled outputs or an updated internal state condition later predictions. Under additive persistent bias and nonexpanding dynamics, an error term can accumulate approximately linearly in horizon; expanding dynamics can amplify perturbations geometrically. Independent zero-mean errors may instead accumulate in root-mean-square scale near under suitable assumptions, while contraction can limit them. One-step error alone therefore does not determine a rollout-error curve.
Two models with similar one-step error can have different long-horizon behavior because their errors differ in bias, correlation, state sensitivity, or exposure to self-generated inputs. Bias is one mechanism, not the only one. The corridor experiment therefore reports recursive RMSE against horizon rather than inferring rollout quality from teacher-forced error.
A second issue is multimodal futures. A state near an unstable choice, or an observation that hides relevant state, can support several plausible outcomes. A stochastic generator can represent multiple samples, but a diffusion, flow, or autoregressive architecture does not guarantee calibrated mode probabilities. A deterministic predictor may return a mean trajectory that corresponds to none of the outcomes. Calibration must be measured against repeated outcomes or a trusted reference distribution.
Occlusion memory. When a task requires tracking a hidden object, its reappearance should be consistent with prior observations and intervening actions, within uncertainty. A generator that always restores the last visible position can fail this test even if the visible background stays coherent. IntPhys (Riochet et al., 2018) evaluates physical plausibility and object-permanence expectations in videos; this chapter's controlled reappearance test asks a narrower, action-conditioned question.
An occlusion probe is useful because the current pixels do not reveal the object's location. A frame generator may still use temporal context or learned regularities; it is not forced to interpolate or freeze. Conversely, an explicit latent state may still track the wrong trajectory. Compare reappearance positions after controlled hidden intervals and actions, and include a simple velocity-extrapolation baseline. The aggregate visibility split in the toy below is only a diagnostic, not an isolated memory test.