What Is a World Model? › Where World Models End: Generative Models and Simulators

"World model" sits between two neighboring concepts with which it is constantly confused: generative models on one side and simulators on the other. Drawing both boundaries sharply will clarify a great deal of contemporary debate. Both confusions arise from real overlap: generative models and world models both model distributions over futures, and simulators and world models both answer action-conditioned queries. The boundaries lie not in what these systems compute but in what they are for and what they guarantee.

A generative model captures the distribution of data: images, audio, text, or video. Modern video generators produce startlingly plausible clips, and clips of falling objects, flowing water, and billowing smoke exhibit many regularities we associate with physical intuition. Does that make a video generator a world model?

By our operational definition, usually not yet, for three reasons.

First, action-conditioning. A video model samples $p(\text{future frames} \mid \text{past frames})$. A world model must answer $p(\text{future} \mid \text{past}, a)$: what changes because of what I do. Without an action channel, the model can extrapolate passive dynamics but cannot support decisions. Interactive generative models that accept latent actions, such as the Genie line of work, are explicitly designed to close this gap, and we study them in Part IX. The gap is deeper than adding an input slot: conditioning on actions requires training data in which actions were actually taken and their consequences observed, which means interaction data, teleoperation logs, or simulator rollouts, none of which passive internet video provides.

Second, controllability and consistency. A world model must respond coherently and persistently to interventions: if the agent picks up a cup, the cup should stay picked up in all subsequent predictions. Purely correlational video models can "forget" object permanence over long horizons because nothing in their training objective forces consistency under intervention. Causal structure, not just statistical fidelity, is at stake here, and Part IV treats causality in world models in depth. Consistency failures are subtle because they are invisible in short clips: a two-second generation can look flawless while containing objects that flicker, merge, or teleport across cuts, defects that only matter to an agent trying to maintain a coherent plan across minutes.

Third, decision sufficiency. Even an action-conditioned, consistent video model is a world model only relative to tasks whose decisions it improves. A gorgeous renderer with subtly wrong contact dynamics may produce beautiful videos while misleading a robot into crushing every object it grasps. Contact dynamics are the canonical example because they are visually subtle and mechanically decisive: a millimeter of penetration error is undetectable to the eye and catastrophic to a gripper.

The honest summary is that the boundary is a gradient, not a wall. Action-conditioned generative models trained on interaction data increasingly qualify as world models; passive ones do not. What converts a generative model into a world model is not architectural glamour but the operational test: conditioning on actions, consistency under intervention, and usefulness for control. Expect the gradient to shift over time as interactive data becomes more abundant, and expect the debate to continue, because the philosophical register asks questions the operational test cannot settle.

At the other end sits the simulator: MuJoCo, Isaac Gym, game engines, flight simulators, digital twins. A simulator certainly predicts action-conditioned futures, so is it a world model? Under the operational definition, yes: it is a world model whose dynamics were authored rather than learned. The interesting question is what distinguishes the two origins, because the differences drive methodology across this entire handbook.

Key contrasts between learned world models and engineered simulators:

• Source of dynamics: A simulator's rules are hand-coded from physical theory or game design. A learned world model's rules are estimated from data, inheriting the biases and gaps of that data.
• State access: A simulator exposes the true underlying state directly, and can be paused, inspected, and reset to arbitrary states. A learned model typically works in a latent space with no guaranteed correspondence to physical quantities, and cannot be reset to an arbitrary ground-truth configuration.
• Uncertainty: A simulator is (within its domain) confidently wrong or confidently right; it expresses no doubt. A learned model can and should represent its own uncertainty, which enables cautious exploration and risk-sensitive planning.
• Fidelity guarantees: Simulator fidelity is audited against requirements and validated by engineers. Learned model fidelity is statistical, holding only near the training distribution, and silently degrading outside it.
• Cost profile: Simulators are cheap to query but expensive to build and calibrate; learned models are expensive to train but amortize across tasks and can capture realities too messy to hand-code.

Each weakness of one is a strength of the other, which explains the two great hybrid strategies of the field. Sim-to-real uses engineered simulators as training grounds and transfers policies to reality despite the inevitable mismatch. Learning from real data builds world models where no adequate simulator exists, such as deformable objects, biology, finance, and the open web. And the two strategies converge in modern practice: learned world models trained inside simulators, simulators calibrated against real data, and foundation-scale models distilling both. Part XII on reliable world models confronts the failure modes of each approach. The convergence is worth watching as a trend: the historical split between "authored worlds" and "learned worlds" is dissolving into a continuum of calibrated hybrids, where each side patches the other's weaknesses.

It is worth naming the full spectrum explicitly, because individual systems populate every rung:

• Lookup tables and memorized transitions: a model that has only seen exactly the situation before.
• Parametric regressors: linear models, Gaussian processes, small networks fitting $(s, a) \to s'$.
• Latent dynamics models: neural encoders and recurrent transitions over learned state spaces, the architecture family of Part V.
• Action-conditioned generative models: pixel- or token-space predictors of futures under actions.
• Engineered simulators: authored dynamics with privileged state access.

Nothing about this ladder implies that higher is better. A well-fitted linear model around an equilibrium point flies aircraft and balances robots every day, while a magnificent video generator remains unable to decide anything. Fitness for purpose, again, is the only ranking that matters. Treat the ladder as a menu of complexity levels, ordered by expressive power, and choose the lowest rung that satisfies the decision-sufficiency requirement for your task. Complexity you do not need is not neutral: it costs data, compute, interpretability, and failure modes.