Stochasticity and Uncertainty

Imagine a robot waiting at an intersection, watching a car approach the light. One question is what will the car do: brake, or continue through the yellow? Another second of observation may reveal braking, steering, or signaling, but the robot's current sensors and state representation may still omit variables that influence the decision. A second question is how well do we understand the situation: have we seen enough intersections like this one to trust our dynamics model at all? Perhaps our model was trained in a different city, or only on daylight traffic rather than midnight rain. These are different kinds of not-knowing. Residual spread in a specified data-generating conditional law, after fixing the information set and treating that law as known, is aleatoric relative to that representation; posterior-predictive spread caused by uncertainty about the law is epistemic. Uncertainty about whether the learned model applies here is epistemic; a better-specified model or additional informative data can reduce it. Confusing them leads to bad engineering because the remedies differ. Aleatoric spread calls for a suitable distribution, while epistemic uncertainty calls for better evidence, better assumptions, or a decision to stop trusting the model.

A point prediction can average across separated futures and, by itself, cannot expose whether its error comes from outcome noise or model uncertainty. Auxiliary diagnostics can help, but the point value alone has the same format in familiar and unfamiliar situations. A planner that treats a low-density conditional mean as the realized next state can then optimize for an outcome the transition scarcely supports.

This chapter develops the vocabulary and machinery for representing both kinds of uncertainty in learned world models. We will see when point predictions lose important structure, how unresolved contact branches can create multimodal futures, how ensembles and Bayesian methods approximate model uncertainty, and how uncertainty can change through long rollouts. Throughout, we keep one theme in view: an uncertainty estimate earns its place when a downstream decision procedure can use it and when its claimed probabilities are empirically checked.

Earlier chapters used transition models of the form $s_{t+1} = f(s_t, a_t)$, a function that maps state and action to next state. Deterministic environments are valid special cases, representable by a transition distribution concentrated at $f(s_t,a_t)$. When unresolved state, stochastic forcing, sensors, or other agents matter, the more general object is a conditional distribution from which $s_{t+1}$ is drawn:

where:

• $s_t$: the state of the world at time $t$
• $a_t$: the action taken by the agent at time $t$
• $s_{t+1}$: the state at the next time step, now a random draw rather than a deterministic output
• $p(s_{t+1} \mid s_t, a_t)$: the conditional transition distribution over the chosen state representation; it is Markov-complete only when the conditioning state is sufficient

Notice what this notation changes conceptually. The same measured state and action can lead to different outcomes. Repeated differences falsify a deterministic transition only relative to the variables measured and controlled; hidden state, measurement error, uncontrolled inputs, and nonstationarity remain alternative explanations.

The randomness in this distribution has several distinct sources, and it is worth naming them because they demand different responses:

• Aleatoric uncertainty relative to the chosen representation: unresolved contact branches, sensor noise, other agents' decisions, or turbulent forcing can remain random after conditioning on the variables the model receives. More repetitions do not remove that conditional spread; the model must represent it.
• Hidden state: the transition looks random only because the state is not sufficient. As we discussed in the chapter on sufficient state, apparent noise can reflect a missing variable, and the causal framing of the previous chapters tells us when that is the case. In a constructed door model where hinge angle is the only omitted determinant, adding it to the state makes the open-versus-jam transition deterministic. The correct response is to find and include the missing variable.
• Model ignorance: the true dynamics may be effectively deterministic, but our model class cannot represent them, or our data does not identify the parameters. A better-specified model or additional informative data can reduce this uncertainty, although misspecification and non-identifiability can prevent it from vanishing.

These sources are blended in finite datasets. The distinction matters because informative data can constrain model ignorance, richer state can explain apparent noise, and an expressive distribution can represent spread that remains after conditioning. The categories are therefore relative to the model's information set, not immutable labels attached to a physical phenomenon.

Assuming a finite conditional second moment, population squared-error risk is minimized pointwise by the conditional mean $\mathbb{E}[s_{t+1} \mid s_t, a_t]$. If optimization reaches the global population-risk minimum within a restricted function class, the result is that class's best $L^2$ approximation. Finite samples, regularization, or imperfect optimization can move the fitted regressor away from that best-in-class target. The conditional mean can still lose branch identity when the predictive distribution has separated modes. Consider a ball that deflects left or right around a wedge. Its conditional mean can lie in a low-density gap between the two branches, even though neither branch passes through that location.

The figure below shows a one-dimensional construction: sampled next states concentrate at two separated deflections, while the empirical mean lies in a gap where this finite sample contains no outcome.

A planner that treats this mean as the realized next state can optimize for the low-density gap rather than either likely branch. Robustness, replanning, or explicit uncertainty can prevent that mistake.

A single predicted trajectory contains no internal decomposition of extrapolation, model uncertainty, and outcome noise. Two identical point outputs can therefore have different epistemic status. External residual and out-of-distribution diagnostics can add information, while uncertainty estimates are one important aid when decisions query poorly covered regions.

The literature splits uncertainty into two categories, and the distinction does real work once you start designing and debugging world models. It is not taxonomic decoration. It determines what you do next when a model misbehaves, what kind of data you collect, and what guarantees you can claim.

The terms trace to Latin alea, a game of dice or chance, and Greek episteme, knowledge. The practical distinction is conditional: for a known fair coin, another flip remains fifty-fifty however many earlier flips we observe, while uncertainty about an unknown bias can shrink as informative flips accumulate.

We can make this concrete with the coin itself. The figure below shows the posterior density over an unknown bias after observing ten flips and after one hundred more, computed with a uniform prior. In this construction, the posterior becomes more concentrated around the observed frequency. What shrinks is uncertainty about the bias; conditional on a fixed bias $p$, a new flip remains Bernoulli$(p)$.

In a Bayesian treatment, the conditional transition and parameter posterior are factors in the posterior-predictive integrand; marginalizing their product yields the predictive distribution. Let $\theta$ denote the parameters of a transition model $p_\theta(s_{t+1} \mid s_t, a_t)$, and let $\mathcal{D}$ be the collected transition data. Rather than one model, Bayes maintains a posterior over models, weighting each parameter setting by how well it explains the observed data:

where:

• $\theta$: the parameters of the transition model
• $\mathcal{D}$: the collected transition data
• $p(\mathcal{D} \mid \theta)$: the likelihood, how well parameters $\theta$ explain the observed transitions
• $p(\theta)$: the prior belief over parameters before seeing data
• $\propto$: proportionality, the normalizing constant $p(\mathcal{D})$ is omitted because it does not depend on $\theta$

The posterior combines the likelihood, which scores how well each parameter setting predicts the observations, with the prior. Under identifiability, suitable regularity, an adequate model, and informative data, the posterior can concentrate as data accumulate. Misspecification, weak identification, or redundant observations can preserve prior sensitivity and multiple plausible parameter settings.

The posterior predictive distribution over next states marginalizes over that posterior:

where:

• $p(s_{t+1} \mid s_t, a_t, \theta)$: the within-model predictive distribution at fixed $\theta$; calling its residual spread aleatoric assumes $\theta$ and the conditioning variables contain the learnable structure relevant to that distinction
• $p(\theta \mid \mathcal{D})$: the posterior weight given to that model given the data
• $d\theta$: the integration runs over all plausible parameter settings

The intuition is a weighted average: each candidate model contributes its own prediction, weighted by its posterior plausibility. The inner term carries within-model predictive spread, while the outer integral carries uncertainty across parameter settings. With enough informative data and an identifiable, adequate model, the posterior may concentrate. Where information is weak, a sound uncertainty method may widen if its plausible models predict differently; strong shared priors or a collapsed approximation can instead remain narrow.

For a scalar next state, or componentwise for a vector state, the predictive variance decomposes exactly by the law of total variance:

where:

• $\mathrm{Var}[s_{t+1} \mid s_t, a_t, \mathcal{D}]$: the total predictive variance under the posterior predictive distribution
• $\mathbb{E}_{\theta}[\cdot]$: expectation taken over the posterior $p(\theta \mid \mathcal{D})$
• $\sigma^2_\theta(s_t, a_t)$: the within-model conditional variance predicted by model $\theta$ at this state and action
• $\mathrm{Var}_{\theta}[\cdot]$: variance across models, again under the posterior
• $\mu_\theta(s_t, a_t)$: the mean prediction of model $\theta$

For a full vector-state covariance, use $\mathrm{Cov}(Y)=\mathbb{E}_\theta[\mathrm{Cov}(Y\mid\theta)]+\mathrm{Cov}_\theta(\mathbb{E}[Y\mid\theta])$.

The first term averages each model's own conditional variance, and the second measures disagreement among conditional means. The law of total variance guarantees that this split of predictive variance is exact. The second term captures parameter uncertainty in the conditional mean, not every parameter-driven difference: $\theta$ can also affect conditional variance or distributional shape, and variance alone cannot encode all shape uncertainty. If every plausible model made the same mean prediction, the variance of the means would be zero even when the models differed in other aspects of their predictive distributions.

This decomposition is more than bookkeeping. It tells you what different diagnostics measure. A single heteroscedastic model at fixed parameters estimates within-model conditional variance. Spread among ensemble means approximates the between-model mean-variance term. Together those terms characterize total predictive variance under the stated hierarchy, not the full predictive uncertainty distribution. Conflating the diagnostics still causes predictable mistakes: a model trained with a homoscedastic Gaussian head reports the same confidence everywhere, including in regions it has never seen, and an ensemble with deterministic members reports zero uncertainty about the amount of noise in a contact event even when its members disagree wildly about where the object will land. Each diagnostic is blind to the uncertainty the other one sees.

A useful diagnostic habit: when a world model behaves badly, ask which kind of uncertainty it may be missing. If it averages over distinct outcomes, a richer output family such as mixtures, latent variables, or diffusion sampling may address a representational failure. If it is confidently wrong in a new regime, ensembling, Bayesian inference, or active data collection may expose or reduce model uncertainty. These are candidate remedies to validate, not categorical diagnoses. The remedies differ, and we will cover both. Misdiagnosis wastes effort in a characteristic way: adding mixture components to a model whose problem is epistemic produces a model that is confidently wrong in a richer format, and adding ensemble members to a model whose problem is aleatoric averaging produces several models that all average over the same impossible middle.

One honest caveat: the boundary between the two categories is drawn relative to your state representation and model class. Friction that you have not modeled shows up as aleatoric noise at your level of description, but a richer model with a friction parameter would move it into the epistemic column, where more data helps. The coin with the unknown bias is aleatoric at the level of "flip outcomes given a fixed bias model" and epistemic at the level of "outcomes given that the bias itself is a parameter to be inferred." As we discussed in the chapter on causal models and interventions, deciding what counts as exogenous randomness is itself a modeling choice. Apparent aleatoric uncertainty can be latent structure waiting to be found, and discovering that structure is one way world models improve over an agent's lifetime. The practical discipline this suggests is to treat fundamental irreducibility as a hypothesis to be tested rather than a fact to be assumed. If a source can be predicted from an observable you are not yet using, it remains aleatoric relative to the current information set but is not fundamentally irreducible; adding the observable changes the information set and can reduce the residual spread.

A failure mode of a naive stochastic model is getting the shape wrong. A single Gaussian cannot reproduce a conditional density with separated modes, even if it matches the first two moments. More data cannot remove that family mismatch. When branch identity affects a decision, a planner needs those modes rather than only their mean and variance.

Multimodality can arise near contact regimes whose branch selection is unresolved by the model: an object may tip left or right, a key may turn or jam, or a foot may slip or grip. It can also arise when another agent chooses among discrete actions or when several hidden states remain compatible with the same observation history. These examples are multimodal only when multiple branches retain appreciable conditional probability; fully specified contact or policy dynamics can remain deterministic or unimodal.

A classic tool is Bishop's mixture density network (1994). For a scalar next state, instead of outputting a single mean and variance, the network emits the parameters of a Gaussian mixture:

where:

• $K$: the number of mixture components; a component can cover part or all of a mode
• $\pi_k(s_t, a_t)$: the mixing weight of component $k$, produced by a softmax over the components so the weights are positive and sum to one
• $\mathcal{N}(s_{t+1};\, \mu_k, \sigma_k^2)$: a scalar Gaussian over the next state with mean $\mu_k(s_t, a_t)$ and variance $\sigma_k^2(s_t, a_t)$; a vector state instead requires a covariance parameterization such as an isotropic, diagonal, or full covariance

Together, the components can represent several modes, and the mixing weights describe component mass. Components are exchangeable and need not correspond one-to-one with physical branches: they may overlap, split a mode, or collapse onto the same mode. Maximum likelihood permits mode coverage but does not guarantee it, so initialization, capacity, optimization, and validation all matter.

Other architectures offer different flexible parameterizations of multimodal distributions, while still imposing their own finite-model constraints:

• Categorical latent variables: sample a discrete latent $z \sim p(z \mid s_t, a_t)$, then condition the decoder on $z$. The decoder may itself be deterministic or stochastic. The latent can encode branch choice, but its categories are not guaranteed to match physical modes.
• Autoregressive token models: factor the joint distribution over future observations into a product of conditionals over discrete tokens, and let the chain structure express complex multimodality compositionally. We will develop this in the chapter on autoregressive token world models. For a length-$T$ chain over a fixed vocabulary of $V>1$ token values, the Cartesian product contains $V^T$ possible token sequences, although those sequences need not decode to $V^T$ physically distinct futures or receive useful probability mass.
• Diffusion and flow models: transform a simple noise distribution into samples through learned dynamics. Diffusion models define iterative generative processes and are commonly trained with variational-bound or denoising objectives; normalizing flows provide explicit change-of-variables densities when their transforms are invertible. Both can represent complex high-dimensional multimodality, subject to architecture, training, and sampling limits.

For separated multimodal futures, preserve the modes or sample individual branches rather than substituting a low-density mean. The worked example below shows this failure for a deliberately bimodal construction.

How should a practitioner produce these distributions? Distributional heads, empirical ensembles, and Bayesian approximations make different cost and modeling choices. They are not interchangeable: a distributional likelihood addresses within-model outcome spread, while ensembles and Bayesian approximations target uncertainty over fitted functions or parameters.

For a scalar target, let the network emit $\mu_\theta(s_t, a_t)$ and $\ell_\theta(s_t, a_t)=\log \sigma_\theta^2(s_t, a_t)$, and set $\sigma_\theta^2=\exp(\ell_\theta)>0$ before evaluating the Gaussian negative log likelihood, up to the additive constant $\tfrac{1}{2}\log(2\pi)$ per observation:

where:

• $s_{t+1}$: the observed next state in the training data
• $\mu_\theta(s_t, a_t)$: the network's predicted mean for that transition
• $\sigma^2_\theta(s_t, a_t)$: the network's predicted variance, which is a function of the state and action
• $\sum_t$: the sum over all transitions in the training set

For a vector state, this expression applies componentwise only under a diagonal conditional covariance; a full-covariance Gaussian uses a quadratic form and a log-determinant.

The first term is a squared error scaled by the predicted variance, and the second prices large variances. Without it, a model could send $\sigma^2$ to infinity and drive the scaled residual toward zero. The fitted likelihood trades variance against observed residuals; it does not prove that a learned large variance is causally irreducible noise.

The state- and action-dependent variance represents heteroscedastic uncertainty. A single global variance is misspecified when conditional noise varies with state or action, but is appropriate for genuinely homoscedastic systems. The likelihood lets the model trade residual size against predicted variance, and the logarithmic term prevents unbounded variance from being free. This learned attenuation can improve mean prediction on some tasks, but the result depends on the model, data, and optimization; the loss does not guarantee that high residuals are irreducible noise.

Quantile regression offers an alternative for selected marginals that does not assume a named parametric likelihood: pinball loss estimates selected conditional quantiles, from which intervals can be formed. Pinball fitting alone does not provide distribution-free finite-sample coverage; that guarantee requires additional assumptions or a calibration method such as an appropriate conformal procedure. A finite quantile grid does not specify an arbitrary full distribution, and independently trained quantiles can cross unless monotonicity or rearrangement is enforced. Within those limits, quantiles can represent asymmetric intervals and heavy tails more directly than a symmetric Gaussian head.

Pinball loss makes the asymmetry behind a target quantile explicit. With residual $u=y-q_\tau(x)$, the conventional loss is $\rho_\tau(u)=u(\tau-\mathbf{1}\{u<0\})$. When $\tau=0.1$, positive residuals, outcomes above the prediction and hence underprediction, receive weight $0.1$, while negative residuals receive weight $0.9$; at $\tau=0.9$ the weights reverse. At $\tau=0.5$ they are equal, yielding absolute-error conditional-median regression.

The second family trains $M$ models with controlled diversity, such as independent initializations, bootstrap samples, or shuffled data. Their predictions are empirical ensemble members, not exact posterior samples. Lakshminarayanan, Pritzel, and Blundell (2017) showed strong uncertainty and calibration results on their benchmarks. Training can run in parallel when hardware permits, but total compute and inference cost still grow with $M$. Member spread establishes predictive disagreement; interpreting it as epistemic uncertainty requires a validated diversity construction.

Initialization, resampling, data order, and hyperparameters can lead members to different fitted solutions. Their predictions often agree more in well-covered regions, but coverage does not guarantee agreement, and disagreement need not be caused by data scarcity alone. Because members usually share architecture and representation, they can also share the same extrapolation bias. Ensemble spread is therefore a model-class-dependent proxy, not a guarantee.

With equal-weight Gaussian members, averaging their means and variances and adding the variance of member means implements the corresponding finite-mixture variance identity. A mixture-headed ensemble can represent within-member multimodality and between-member disagreement, but labeling those terms aleatoric and epistemic remains approximate and does not guarantee calibration.

The third family maintains or approximates $p(\theta \mid \mathcal{D})$. Exact inference for general modern neural networks is computationally intractable, although small, discrete, linearized, or conjugate special cases can be tractable. Practical methods trade posterior fidelity for computational cost:

• Variational inference fits a tractable distribution $q(\theta)$, often a diagonal Gaussian, by optimizing the evidence lower bound. A diagonal Gaussian cannot represent parameter correlations; omitting those correlations can materially distort function uncertainty, alongside other limitations such as missing multimodality.
• Laplace approximation fits a local Gaussian around a trained differentiable network using loss curvature. Low-overhead post-hoc variants are available, but a single local Gaussian cannot represent separated posterior modes. Ensembles may span distinct fitted solutions, although they are not exact multimodal-posterior inference.
• Monte Carlo dropout uses dropout-compatible training and repeated stochastic test-time passes under stated variational assumptions. It is simple to add when dropout is already part of the model, but repeated passes have inference cost. See Gal and Ghahramani (2016).
• SWAG and related methods fit a Gaussian to statistics of stochastic-gradient iterates and use it for approximate Bayesian model averaging. See Maddox et al. (2019).

Deep ensembles are a strong empirical baseline. Distributional heads are useful when conditional outcome spread matters, while Bayesian approximations offer alternatives when ensemble cost or posterior structure matters. Some systems, such as PETS, combine probabilistic ensembles with sampling-based propagation. The appropriate layering depends on the uncertainty sources and deployment cost.

A predictive distribution can be sharply peaked and wrong, or wide and correct, or wide and wrong. Calibration ties claimed confidence to observed frequencies under an evaluation design, and downstream decisions often rely on that relationship. Miscalibrated probabilities can still carry information, but they do not match observed frequencies as claimed. Material undercoverage can cause a planner that trusts those intervals to underprice risk.

For continuous predictive distributions, outcomes transformed through the predicted CDF form the probability integral transform (PIT). Uniformity is necessary for probabilistic calibration under the usual setup. A U shape often suggests underdispersion, while a central hump often suggests overdispersion, but these shapes are not unique diagnoses and should be checked alongside bias and dependence.

The PIT diagnostic is easiest to trust once seen. The histogram below transforms held-out outcomes through the predicted CDFs of a deliberately overconfident model. Under the usual conditions, a calibrated model yields PIT values that are uniform in the population and an approximately flat held-out histogram up to sampling variation; the U shape here shows too much mass at the extremes, the signature of intervals that are too narrow.

The standard diagnostic is the reliability diagram. Choose nominal coverage levels, compute the empirical fraction of outcomes inside each corresponding predictive interval, and plot empirical against nominal. A curve below the diagonal indicates undercoverage. On the classification benchmarks studied by Guo et al. (2017), modern neural classifiers were often poorly calibrated and held-out logit temperature scaling was an effective post-hoc correction. That result does not establish out-of-distribution regression calibration or justify rescaling predictive variances without separate validation.

Two complementary metrics are worth tracking together:

• Calibration error: how far the selected reliability curve sits from the diagonal. For binned probability calibration, expected calibration error weights each bin's deviation by its frequency; a predictive-interval coverage curve needs its own stated summary across coverage levels.
• Sharpness: how narrow the predictive distributions are, averaged over inputs. Among calibrated models, sharper is better. Assess sharpness alongside calibration because a sharp forecast that materially undercovers can create false confidence.

The pairing matters because each metric is incomplete on its own. A model can achieve some calibration criteria with predictions too diffuse to be useful, and it can look sharp while under-covering outcomes. The practical goal is sharpness subject to the chosen calibration criterion; this is forecast-evaluation guidance, not a consequence of the bias-variance decomposition.

A model can be calibrated on held-out one-step transitions yet miscalibrated on model-generated or policy-induced states. One-step calibration does not transfer automatically to that population. When a model is deployed inside feedback, evaluate calibration on deployment-representative closed-loop rollouts or use sequential methods designed for adaptive data. Part XI on evaluation returns to this in depth.

A one-step predictive distribution is a local quantity. Planning and evaluation often need distributions over $H$-step futures. For complex nonlinear, non-Gaussian models, evaluating the required integrals is usually computationally intractable, so practical methods use approximations. Exact propagation remains possible in linear-Gaussian, finite-state, conjugate, and other special systems.

When one unknown dynamics parameter $\theta$ persists across the rollout, the Bayesian $H$-step predictive distribution integrates that shared parameter outside the trajectory integral. The displayed formula assumes continuous-state densities:

For discrete states, replace the intermediate-state integrals with sums; more generally, interpret the composition as integration of transition kernels against the appropriate state-space measure. where:

• $H$: the rollout horizon, the number of transition steps being composed
• $a_{t:t+H-1}$: the action sequence applied at steps $t$ through $t+H-1$
• $\theta$: one dynamics parameter shared across all $H$ transitions in a sampled model
• $p_\theta(s_{t+h+1} \mid s_{t+h}, a_{t+h})$: the conditional transition under that fixed parameter
• $ds_{t+1} \cdots ds_{t+H-1}$: integration over every intermediate state along the trajectory; only $s_{t+1}$ through $s_{t+H-1}$ are integrated out, since $s_t$ is given and $s_{t+H}$ is the quantity of interest
• $p(\theta \mid \mathcal{D})$: the posterior over the persistent dynamics parameter, integrated once after the full trajectory likelihood is formed

Read the formula as a process: draw one plausible $\theta$ from its posterior, hold it fixed, sample the state transitions, and then average over both intermediate states and plausible parameters. Redrawing $\theta$ independently at every step would define a different model and would remove cross-time dependence induced by the shared parameter. For nonlinear transitions, later distributions can depend on higher-order or multimodal structure, so early shape errors can propagate or amplify; affine-Gaussian or strongly mixing transitions are important counterexamples.

Outside tractable special cases, three common strategies occupy different accuracy-cost tradeoffs:

• Moment matching: propagate only means and covariances, approximating each transition as Gaussian. For a symmetric two-branch mixture, the matched Gaussian has its mode at the low-density middle while preserving a large variance. Later dynamics can propagate, amplify, or erase that approximation error; they do not necessarily inherit it unchanged.
• Particle methods and sampling: represent the future distribution with simulated trajectories that draw transition noise and, when applicable, ensemble members. Sampling can preserve multimodality but may miss modes with too few particles. Under a fixed particle budget, mode-miss risk can worsen when the effective support, branch count, rare-mode probabilities, proposal mismatch, or weight degeneracy worsens with dimension or horizon. Filtering and forward prediction both marginalize trajectories, but they are not the same integral run in reverse.
• Deterministic member propagation: roll each ensemble member forward without within-member noise. The resulting bundle approximates between-model disagreement, not total predictive spread. PETS instead combines probabilistic ensemble members with trajectory sampling.

Uncertainty can compound when recursive prediction enters locally expanding or unstable dynamics. Each state error then perturbs the next input and can be amplified. Contractive dynamics can instead damp errors, and stable stochastic systems may approach a bounded stationary spread. The worked examples below deliberately use settings in which spread grows; they demonstrate a mechanism, not a universal horizon law.

Before the worked example, we can illustrate the mechanism in isolation with a simple linear system whose one-step noise is constant. The figure below shows percentile bands of sampled rollouts: the band is narrow at step one and widens steadily with horizon, even though nothing changes about the per-step noise.

Uncertainty that compounds is not only a problem to mitigate. Several model-based reinforcement-learning methods make planners consume predictive uncertainty:

• Risk-sensitive objectives optimize a risk functional, such as expected utility, an entropic risk, a variance penalty, or a tail statistic. Chance constraints instead bound an estimated failure probability. Their guarantees depend on the predictive distribution, estimator, and optimizer.
• Disagreement penalties discourage candidate plans on which ensemble members disagree. Disagreement is a proxy for model risk, so this heuristic must be validated rather than treated as a trust certificate.
• Trajectory sampling for planning, as in PETS, rolls candidate action sequences through sampled probabilistic dynamics. Sampling represents both uncertainty sources, but robustness also depends on the objective, particle count, model fidelity, and optimization.

Why use a risk functional or chance constraint rather than expectation alone? Two candidate return distributions can have the same expected value while assigning very different probability to a failure tail. An optimizer that sees only expected return treats them as tied; a tail-sensitive objective or explicit failure-probability constraint can distinguish them, provided the predictive distribution and estimator are adequate.

For decisions that depend on branch identity, tails, or constraint violations, held-out one-step mean error is incomplete. A useful world model must represent the aspects of future uncertainty that affect the downstream objective and must be evaluated under that use.

Let us build these ideas in one small, executable study. The mean transition is $f(s, a) = \sin(s) + 0.4a$ with additive noise of standard deviation $0.05$. Fitting a high-degree polynomial only inside a limited interval makes out-of-interval behavior underdetermined and intentionally exposes ensemble disagreement. The construction examines data-dependent disagreement, recursive propagation, a bimodal transition, and empirical interval coverage. Its numerical conclusions apply to this fixed seed, model class, and sampling design.

First, the system and the data.

Each member sees a different bootstrap resample and regularization strength. The regularization jitter adds diversity but is not a calibrated posterior over hyperparameters; it is an explicit experimental perturbation.

We evaluate both ensembles on a grid of states that extends well beyond the training region, with the action held at zero, and summarize each ensemble by its mean and standard deviation across members.

The teal traces are the fitted member functions. Shared data-dependent symmetric-log limits include every shown trace and both ±2-standard-deviation bands with five percent padding; the bands summarize member disagreement rather than calibrated predictive coverage. Outside the sampled state interval, several high-degree polynomial fits diverge while remaining unconstrained by the training loss there. A planner can use that pattern as a warning signal after validating that ensemble spread tracks error for the deployment regime.

Next we iterate each fitted member deterministically from the same initial state under one open-loop action sequence. The resulting spread measures between-member divergence only; it omits the system's within-model noise and therefore is not the total predictive distribution.

Short horizons, feedback, and frequent replanning can reduce exposure to accumulated model error when state estimates are adequate, the closed loop is stabilizing, model and optimization errors remain controlled, and the horizon suits the dynamics. Receding-horizon control replans from the latest state estimate, but it does not reset estimation error, model bias, or optimization failure. Recursive prediction permits error growth in locally expanding regimes; contractive dynamics can damp it.

Now we switch from epistemic to aleatoric structure. In this synthetic transition, an object at state $0.1$ receives an impulse of $\pm 0.6$ with equal probability plus Gaussian noise. The two branches put very little density in the middle, although Gaussian noise gives nonzero density at every finite location. We sample this fixed construction and fit two candidate distributions.

This is the unimodal-distribution trap; its mean also illustrates point averaging. In this construction, the mixture places density near both sampled branches, while the Gaussian mode lies near the middle. Before assuming optimization or data are the only problems, check whether the output family can represent the observed conditional shape.

Finally, we measure empirical coverage on one fixed 300-point test sample spanning the training interval and extrapolation. We compare intervals using the known noise standard deviation with intervals that also add member-mean variance under a Gaussian approximation. Both use the same ensemble mean. The comparison isolates the numerical effect of that added variance term in this construction; it does not establish universal calibration.

The variance sum follows the law of total variance for an equal-weight mixture of members, while converting that variance to Gaussian intervals is an additional approximation. The identity decomposes predictive variance, not every difference in distributional shape. In this seed, the added member variance improves mean absolute coverage error but overcovers at several nominal levels. Bayesian models, quantile regression, and conformal prediction offer different predictive distributions or calibrated sets; they do not all identify the same aleatoric/epistemic decomposition.

Uncertainty-aware probabilistic models contributed to several sample-efficient model-based reinforcement-learning systems. PILCO demonstrated data-efficient policy search with Gaussian-process dynamics and propagated model uncertainty, while PETS combined probabilistic neural ensembles, trajectory sampling, and receding-horizon planning on benchmark tasks. PlaNet uses stochastic latent dynamics for online planning, while Dreamer learns policies through latent imagination. These results establish useful design patterns, not a single causal history or a guarantee that reported uncertainty is calibrated.

The limitations are equally structural, and it is better to internalize them now than to rediscover them in production.

Ensembles quantify disagreement within a chosen construction. Shared architecture, features, and inductive biases can make all members wrong in the same way. Ensembles may show increased uncertainty on some novel inputs, but they may also understate uncertainty when members share extrapolation bias; test this behavior rather than assume it. Classical split-conformal coverage is finite-sample and distribution-free under exchangeability, while adaptive, weighted, and online variants use different assumptions. Closed-loop deployment therefore requires matching the conformal method and evaluation design to the data-generating process.

Aleatoric and epistemic uncertainty are entangled with representation. As we saw, apparent randomness can reflect hidden state. A model can reduce its reported aleatoric uncertainty either by learning real structure or by overfitting noise in a way that happens to fit the training data, and these are hard to tell apart from losses alone. Both mechanisms can lower the fitted training objective; only learning real structure can lower out-of-sample conditional uncertainty by explaining previously unresolved structure. A validation set cannot always distinguish the two, because noise that is unpredictable in the training regime may be predictable in another. This is one more reason the causal perspective of the previous chapters matters: deciding which variables are exogenous noise and which are latent structure is a modeling decision that no likelihood can make for you. The likelihood measures fit under a hypothesis; it cannot tell you whether the hypothesis carved nature at its joints.

Propagation limits depend on the dynamics and approximation. In complex nonlinear, non-Gaussian models, faithful propagation can be costly and cheap approximations can distort shape. Locally expanding dynamics may widen the predictive distribution and weaken action comparisons, while contractive or stable dynamics can bound or reduce spread. Hierarchical models, shorter rollouts, and task-focused latent states are engineering responses, not general guarantees of slower error growth.

Calibration is distribution-dependent. A policy that consumes uncertainty changes which states are visited. IID held-out calibration does not capture this decision-induced shift, so closed-loop or sequential evaluation is needed when the model is deployed adaptively. Avoidance can also reduce data collection in high-uncertainty regions, coupling calibration, exploration, and policy design.

The rest of the book uses these ideas in recurrent state-space models, diffusion-based generators, evaluation protocols, and risk-sensitive planning. Each architecture still needs its uncertainty claims checked under its own data, representation, and deployment distribution.

• World-model transitions are generally conditional distributions; deterministic functions are degenerate cases. Expected squared-error minimization targets the conditional mean, which can fall in a low-density gap for separated multimodal futures.
• Aleatoric uncertainty is conditional spread relative to a specified information set. Epistemic uncertainty reflects limited knowledge and can shrink when informative data and suitable structure identify the model. The boundary depends on representation and assumptions.
• Mixture heads, discrete latents, autoregressive models, and diffusion or flow models offer different ways to preserve multimodal structure. Without semantic supervision or identifiability constraints, these architectures do not guarantee that learned components match physical branches.
• Distributional heads model within-model spread. Ensembles provide empirical between-member disagreement, while Bayesian approximations target parameter or function uncertainty. Their interpretations and calibration require validation.
• Calibration connects claimed probabilities to observed frequencies. For feedback deployment, evaluate on representative closed-loop data or use appropriate sequential methods rather than assuming one-step IID calibration transfers.
• Recursive uncertainty can grow in locally expanding dynamics and can shrink or remain bounded in contractive systems. Moment methods, particles, and member propagation make different approximation and cost tradeoffs.
• Whether apparent noise is hidden structure involves observability, system identification, and sometimes causal assumptions. That question leads naturally to the next chapter's treatment of compositional structure and generalization.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about stochasticity and uncertainty in world models.