Video Generators as Candidate World Models › Summary

Michael BrenndoerferJuly 21, 20262 min read
World Models HandbookAll chapters
Part 9 of 12Foundation and World-Action Models
Chapter 51Video Generators as Candidate World Models › Summary
Reading section 12 of 13 · Summary · 2 minSection 12 of 13 · 2 min

Book contents

World Models Handbook12 parts · 72 chaptersView the whole book →
Active chapter 51 of 72

Section 12 of 13 · 2 min read

Video Generators as Candidate World Models › SummaryOne section · Sections in this chapter12 / 13 ↓
Sections in this chapter13
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

  • An action-conditioned video generator samples pθ(o~t+1:t+K∣ht,at:t+K−1,c)p_\theta(\tilde{o}_{t+1:t+K} \mid h_t, a_{t:t+K-1}, c). A timed, calibrated action channel is needed for the intervention-based planning tests in this chapter; pixels alone do not provide it.
  • Observational conditioning p(ot+1∣ht,at)p(o_{t+1} \mid h_t, a_t) need not equal the interventional quantity p(ot+1∣do⁡(at),ht)p(o_{t+1} \mid \operatorname{do}(a_t), h_t) when action assignment is confounded. Narrow or absent action support instead limits what can be estimated from logged data; it does not by itself prove that the two quantities differ.
  • Perceptual coherence, physical plausibility, and decision usefulness are different properties. Perceptual scores provide partial evidence about appearance; task-relevant constraints and counterfactual probes test specific physical or causal behavior; closed-loop evaluation tests whether the predictions help decisions. None alone certifies the other properties.
  • The qualification ladder has seven rungs: held-out one-step prediction, open-loop rollouts, occlusion-memory probes, matched counterfactual action branches, intervention tests, uncertainty calibration, and closed-loop replanning. Each has its own baseline, false positive, and failure interpretation.
  • A video-space planner can score generated futures with a task cost and estimated uncertainty, optionally filtering candidates by model-estimated risk. Visual inspectability may help, but runtime, perception errors, and model calibration constrain its use.
  • The scalar corridor toy separates action-conditioned and action-blind linear predictors on this policy and simulator. It shows horizon-dependent errors and matched action-branch response; its visible-versus-occluded error grouping is descriptive, not a controlled proof of memory. It makes no direct claim about a named video generator.
  • The remaining frontier is persistent, controllable, low-latency interaction. Generating a convincing clip is not the same as running an environment you can play inside. The next chapter picks up exactly there.