Video Generators as Candidate World Models › Generator Families in One Paragraph Each
World Models HandbookAll chapters Video Generators as Candidate World Models › Generator Families in One Paragraph EachOne section · Sections in this chapter4 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Video generators as world models come in a few architectural flavors. We do not re-derive the algorithms here; earlier chapters cover the mechanics, and only the differences that matter for dynamics matter here.
Autoregressive token video models. A tokenizer maps video into discrete codes, and a dynamics model predicts subsequent codes from prior context. Actions, when available, may enter as tokens, embeddings, or another conditioning signal. Recursive sampling permits extended generation but can compound error. Genie (Bruce et al., 2024) combines a spatiotemporal video tokenizer, an autoregressive dynamics model, and a latent-action model. Its interactive interface belongs to Chapter 52.
Diffusion and flow video models. A model denoises or transports a video representation, sometimes generating a whole clip or block jointly rather than one frame at a time. Sampling cost depends on the number of network evaluations and the video extent processed per evaluation; it is not universally separate frame passes. The Sora report (2024) and UniSim paper (2023) provide examples with different conditioning and evidence. GameNGen (2024) uses diffusion for next-frame game generation and rolls it forward interactively.
Hybrid and interaction-oriented models. These mix representation, dynamics, and rendering choices rather than forming a single latent-state family. GAIA-1 (Hu et al., 2023) predicts discrete video-related tokens autoregressively and uses a diffusion decoder for rendering. IRASim (2024) conditions robot-scene video on actions. DINO-WM predicts visual features for planning, with a separate optional diagnostic decoder; it is not itself a video generator.
The useful comparison is what each system conditions on, what it predicts, how it renders observations, and whether it can roll out under a specified action sequence. Error type cannot be read off the model-family label alone. A tokenized dynamics model may drift in object state or appearance; a diffusion model may also lose identity or miss an action response. Measure these failures directly.
Part IX, Chapter 4, Genie and Interactive Generated Environments, examines latent-action inference and interactive generation in detail. Here we focus on evidence needed before trusting generated video for a specified predictive or control task.