Video Generators as Candidate World Models › Limitations and Impact

Video generators as world models have real potential and real limits, and the two are difficult to disentangle by looking at clips.

Many video generators are trained primarily on observational clips, which need not contain the timed control signals required by a planner. Action-labelled interaction data can help, but its causal value depends on coverage, the behavior policy, confounding, and whether resets or matched interventions are possible. Collecting repeated interventions can be expensive in physical settings; simulation can provide greater control but introduces its own transfer problem. Neither a conditioning token nor a large video corpus alone resolves these issues.

The second limitation is compute. Sampling many high-dimensional futures within a control deadline may be costly, but latency depends on architecture, sampler, candidate count, horizon, hardware, and batching. A 10 Hz controller has a 100 ms cycle; whether a particular denoising budget fits that cycle requires measurement on its target hardware. Latent rollouts may avoid repeatedly decoding full frames, while video rollouts expose details that a compressed state might discard. These are design choices to benchmark against the task's latency and information requirements, not a fixed speed hierarchy.

The third limitation is diagnostic coverage. FVD, PSNR, and LPIPS measure different aspects of generated clips; none is designed to establish correct action effects or task success. Their sensitivity to a small but decisive event depends on the event, representation, aggregation, and metric. Pair perceptual scores with targeted state/event checks, matched interventions where feasible, and closed-loop outcomes for planning claims.

Each diagnostic has a baseline and failure mode. A generator may look calibrated on coarse event bins while missing conditional structure; a planner may exploit simulator error or an evaluation artifact. The ladder is a scaffold, not a proof or a requirement to pass all seven tests. Choose the rungs that match the intended use, disclose their limits, and test policy and scene shifts when relevant. Parts X and XI develop applications and evaluation questions.

Rendered futures can make some errors easy to spot: an object may vanish across an occlusion or a predicted contact may look wrong. Latent models are not inherently opaque, though; probes, decoded predictions, and decision tests can reveal errors that a clip review misses. Visual inspection is a useful diagnostic alongside quantitative and interventional evaluation. Part IX, Chapter 4 turns to interactive generated environments and latent-action interfaces.