Taxonomies of World Models › Deterministic and Stochastic Dynamics

Independently of the state format, every model must decide how to represent the spread of possible futures. The world is noisy: sensors jitter, contacts slip, other agents act unpredictably. The taxonomy here separates models that output one future from models that output a distribution. This axis is orthogonal to the state axis in principle, though in practice certain combinations dominate: pixel and latent models appear in both deterministic and stochastic forms, while symbolic models almost always commit to single outcomes because their rule-based semantics leave little room for graded belief.

A deterministic model computes $\hat{s}_{t+1} = f_\theta(s_t, a_t)$. Training is typically squared-error regression, inference is a single forward pass, and rollouts are cheap and differentiable. Most of the systems in this handbook, including the one we build below, are deterministic. Determinism is not a lazy default: for many controlled systems, especially those with strong feedback stabilizing the dynamics, the spread of plausible futures at the timescales that matter is small enough that a single committed trajectory is an adequate summary, and the computational savings are substantial.

The blind spot appears whenever the true transition is stochastic. Squared-error training drives a deterministic model toward the conditional mean of the next state. When the future is unimodal, the conditional mean is a fine summary. When it is multimodal, the mean is a fiction: a state that lies between two plausible outcomes and may itself be physically impossible. A deterministic model of a car approaching a fork does not predict "left or right"; it predicts "into the median." The figure below makes the pathology concrete with a minimal bimodal transition: the conditional mean sits in a valley between the two modes, a value the environment essentially never produces, yet squared-error regression commits to exactly that value.

Worse, during multi-step rollouts the model treats its own fictional outputs as facts, committing to a single imagined future with no indication of how fragile it is. Nothing in the model's output distinguishes a prediction it is sure of from one it is guessing at, so a downstream planner weights them equally.

This failure has a subtle training-time cousin. Under noise, regression pushes the model to predict expectations, and a controller optimizing against those expectations systematically misestimates the value of risky options, a phenomenon related to the optimism and exploration pathologies we examine in Part VII. The mechanism is worth spelling out because it is counterintuitive: if action A yields +10 or -10 with equal probability and action B yields 0 for certain, the conditional-mean model evaluates both as 0, erasing the difference entirely. A risk-seeking planner loses the upside information; a risk-averse planner loses the downside warning. Either way, the information needed to distinguish the actions was destroyed at training time, not at decision time, which makes it unrecoverable by better planning on top of the same model.

Stochastic models instead learn a conditional distribution $p_\theta(s_{t+1} \mid s_t, a_t)$ and answer queries by sampling. There are several standard mechanisms, differing in what uncertainty they capture:

• Explicit distribution heads: the network outputs the parameters of a chosen family, such as a Gaussian mean and variance, a mixture of Gaussians for multimodality, or a categorical over discrete outcomes. Sampling is direct; likelihoods are computable. The choice of family is itself a modeling commitment: Gaussians assume unimodal spread, mixtures restore multimodality at the price of harder optimization, and categorical heads suit inherently discrete outcomes.
• Latent noise: the model injects randomness into its hidden state at every step, as Dreamer does with its stochastic latent variables, so that diversity accumulates naturally over rollouts. Rather than predicting a distribution over next states directly, the model predicts a distribution over a latent variable that modulates the transition, which keeps individual samples sharp while allowing the ensemble of samples to fan out.
• Deep ensembles: several independently trained deterministic models vote. Their disagreement is a practical proxy for epistemic uncertainty, the uncertainty coming from limited data, though it says little about inherent randomness. Ensembles are popular because they require no change to the base model, only repeated training, and because their disagreement tends to grow in exactly the regions where training data was sparse.
• Generative samplers: diffusion or flow models conditioned on history and action produce high-fidelity samples of complex futures such as images, at the cost of expensive iterative sampling. These are the current frontier for stochastic imagination over high-dimensional observations, trading compute at inference time for expressive power that parametric families cannot match.

Sampling-based models can do things deterministic ones cannot. Planners can propagate uncertainty through rollouts and choose actions that are robust across futures rather than optimal in an average fiction. Evaluation can ask not just "was the prediction accurate?" but "was the model's confidence calibrated?", which separates predictive fidelity from uncertainty quality, two properties that neither implies the other. And rare-event reasoning becomes possible: you can sample until the dangerous branch appears, instead of never seeing it at all. For safety-critical applications this last capability is often the decisive argument, since the events that matter most are precisely the ones a conditional-mean model smooths away.

The costs are real. Stochastic rollouts require many samples to estimate anything, multiplying compute. Training with likelihood objectives can be unstable, especially with variance parameters that can collapse or explode. And calibration is notoriously difficult: a model can be stochastic in form while badly miscalibrated in fact, spreading probability in ways that match neither the data nor honest ignorance. Overconfident stochastic models are arguably worse than honest deterministic ones, because they lend an air of statistical rigor to guesses, and downstream consumers tend to trust outputs wearing the costume of probability. Calibration assessment therefore belongs in any serious evaluation protocol, and Part XI treats it as a first-class metric rather than an afterthought.

Chapter 3 introduced the qualification test for world models, and one of its central lessons bears repeating here because the deterministic/stochastic choice interacts with it directly. Multi-step prediction error compounds: each predicted state becomes the input to the next prediction, so small one-step errors feed on themselves. Deterministic models compound toward a single confident wrong trajectory. Stochastic models compound toward a widening cloud, which is more honest but also harder to plan with, since the cloud eventually swallows any signal about distant consequences. Neither choice eliminates compounding; they only determine its shape. Mitigation strategies, such as short-horizon training, closed-loop fine-tuning, and uncertainty-aware replanning, occupy much of Part VI, Part VII, and Part XII.

There is a classical echo here worth hearing. Feedback control has always been the ultimate defense against model error: rather than demanding a perfect open-loop prediction over a long horizon, classical schemes re-predict at every step from freshly measured state, so errors are corrected before they accumulate. Model-predictive control formalizes this as receding-horizon planning, and the same logic transfers unchanged to learned world models. When you see a learned model's long-horizon rollouts degrade, the practical response inherited from control theory is not necessarily a better model but a shorter horizon plus faster replanning, letting measurement close the loop that imagination cannot hold open alone.