Stochasticity and Uncertainty › Limitations and Impact

Uncertainty-aware probabilistic models contributed to several sample-efficient model-based reinforcement-learning systems. PILCO demonstrated data-efficient policy search with Gaussian-process dynamics and propagated model uncertainty, while PETS combined probabilistic neural ensembles, trajectory sampling, and receding-horizon planning on benchmark tasks. PlaNet uses stochastic latent dynamics for online planning, while Dreamer learns policies through latent imagination. These results establish useful design patterns, not a single causal history or a guarantee that reported uncertainty is calibrated.

The limitations are equally structural, and it is better to internalize them now than to rediscover them in production.

Ensembles quantify disagreement within a chosen construction. Shared architecture, features, and inductive biases can make all members wrong in the same way. Ensembles may show increased uncertainty on some novel inputs, but they may also understate uncertainty when members share extrapolation bias; test this behavior rather than assume it. Classical split-conformal coverage is finite-sample and distribution-free under exchangeability, while adaptive, weighted, and online variants use different assumptions. Closed-loop deployment therefore requires matching the conformal method and evaluation design to the data-generating process.

Aleatoric and epistemic uncertainty are entangled with representation. As we saw, apparent randomness can reflect hidden state. A model can reduce its reported aleatoric uncertainty either by learning real structure or by overfitting noise in a way that happens to fit the training data, and these are hard to tell apart from losses alone. Both mechanisms can lower the fitted training objective; only learning real structure can lower out-of-sample conditional uncertainty by explaining previously unresolved structure. A validation set cannot always distinguish the two, because noise that is unpredictable in the training regime may be predictable in another. This is one more reason the causal perspective of the previous chapters matters: deciding which variables are exogenous noise and which are latent structure is a modeling decision that no likelihood can make for you. The likelihood measures fit under a hypothesis; it cannot tell you whether the hypothesis carved nature at its joints.

Propagation limits depend on the dynamics and approximation. In complex nonlinear, non-Gaussian models, faithful propagation can be costly and cheap approximations can distort shape. Locally expanding dynamics may widen the predictive distribution and weaken action comparisons, while contractive or stable dynamics can bound or reduce spread. Hierarchical models, shorter rollouts, and task-focused latent states are engineering responses, not general guarantees of slower error growth.

Calibration is distribution-dependent. A policy that consumes uncertainty changes which states are visited. IID held-out calibration does not capture this decision-induced shift, so closed-loop or sequential evaluation is needed when the model is deployed adaptively. Avoidance can also reduce data collection in high-uncertainty regions, coupling calibration, exploration, and policy design.

The rest of the book uses these ideas in recurrent state-space models, diffusion-based generators, evaluation protocols, and risk-sensitive planning. Each architecture still needs its uncertainty claims checked under its own data, representation, and deployment distribution.