Video Generators as Candidate World Models › What Video Generation Adds (and What It Does Not)
World Models HandbookAll chapters Video Generators as Candidate World Models › What Video Generation Adds (and What It Does Not)One section · Sections in this chapter3 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Rendering the future as video has a genuine advantage and a genuine hazard.
The advantage is inspectability. Part IX, Chapter 2's feature-space rollouts require a probe or decoder for visual inspection. Generated frames expose visible events such as an object leaving a target region or an arm intersecting a clearly visible wall. A frame-based check still needs an appropriate viewpoint and, for automated scoring, a validated perception procedure. Hidden state, forces, and ambiguous contact cannot be read reliably from the clip alone.
The hazard is that visual plausibility and predictive correctness are different tests. A good perceptual score can coexist with a failed action-response or dynamics probe. In the occluded-corridor example, a smooth clip could reintroduce the object at a position inconsistent with the specified actions. Modern video generators may model whole clips and long-range temporal structure—not just isolated frames—yet their training objectives do not automatically enforce the action-conditioned physical constraints of a particular task. The Sora technical report discusses both emerging simulation-like behavior and explicit failure modes.
The property that a clip resembles a plausible continuous recording: coherent motion, lighting, textures, and object appearance. Paired-frame scores such as PSNR, SSIM, and LPIPS and video-distribution scores such as FVD each probe different aspects of visual quality; none alone isolates temporal coherence or action fidelity.
The property that the model's internal state at time , together with the planned action sequence, is enough to predict task-relevant futures correctly. Sufficiency is always relative to a task and a policy class. A latent state that suffices for "where is the object?" may not suffice for "will the robot's gripper collide with the wall?"
A useful way to organize the analysis is to ask three questions of any video generator:
- Does it look right? Perceptual coherence on held-out clips.
- Does it behave right? Contact geometry, inertia, occlusion memory, object identity, and count preservation.
- Does it help decide right? Uncertainty-aware planning quality in closed loop.
The first question can use perceptual metrics and human inspection, but not one universal score. The second needs probes that isolate a physical or causal property. The third tests the intended decision use in a control loop against baselines. A model can look good but fail a dynamics probe; it can also predict well yet fail in closed loop because of planner search, task-cost design, latency, or model exploitation. Diagnose those components before blaming the predictor.
The gap between visual quality and control utility is an empirical question for each model and task, not a prevalence claim about the field.