Stochasticity and Uncertainty › Uncertainty Propagation Through Rollouts

A one-step predictive distribution is a local quantity. Planning and evaluation often need distributions over $H$-step futures. For complex nonlinear, non-Gaussian models, evaluating the required integrals is usually computationally intractable, so practical methods use approximations. Exact propagation remains possible in linear-Gaussian, finite-state, conjugate, and other special systems.

When one unknown dynamics parameter $\theta$ persists across the rollout, the Bayesian $H$-step predictive distribution integrates that shared parameter outside the trajectory integral. The displayed formula assumes continuous-state densities:

$$p(s_{t+H} \mid s_t, a_{t:t+H-1}, \mathcal{D}) = \int p(\theta \mid \mathcal{D}) \left[ \int \cdots \int \prod_{h=0}^{H-1} p_\theta(s_{t+h+1} \mid s_{t+h}, a_{t+h}) \, ds_{t+1} \cdots ds_{t+H-1} \right] d\theta.$$

For discrete states, replace the intermediate-state integrals with sums; more generally, interpret the composition as integration of transition kernels against the appropriate state-space measure. where:

• $H$: the rollout horizon, the number of transition steps being composed
• $a_{t:t+H-1}$: the action sequence applied at steps $t$ through $t+H-1$
• $\theta$: one dynamics parameter shared across all $H$ transitions in a sampled model
• $p_\theta(s_{t+h+1} \mid s_{t+h}, a_{t+h})$: the conditional transition under that fixed parameter
• $ds_{t+1} \cdots ds_{t+H-1}$: integration over every intermediate state along the trajectory; only $s_{t+1}$ through $s_{t+H-1}$ are integrated out, since $s_t$ is given and $s_{t+H}$ is the quantity of interest
• $p(\theta \mid \mathcal{D})$: the posterior over the persistent dynamics parameter, integrated once after the full trajectory likelihood is formed

Read the formula as a process: draw one plausible $\theta$ from its posterior, hold it fixed, sample the state transitions, and then average over both intermediate states and plausible parameters. Redrawing $\theta$ independently at every step would define a different model and would remove cross-time dependence induced by the shared parameter. For nonlinear transitions, later distributions can depend on higher-order or multimodal structure, so early shape errors can propagate or amplify; affine-Gaussian or strongly mixing transitions are important counterexamples.

Outside tractable special cases, three common strategies occupy different accuracy-cost tradeoffs:

• Moment matching: propagate only means and covariances, approximating each transition as Gaussian. For a symmetric two-branch mixture, the matched Gaussian has its mode at the low-density middle while preserving a large variance. Later dynamics can propagate, amplify, or erase that approximation error; they do not necessarily inherit it unchanged.
• Particle methods and sampling: represent the future distribution with simulated trajectories that draw transition noise and, when applicable, ensemble members. Sampling can preserve multimodality but may miss modes with too few particles. Under a fixed particle budget, mode-miss risk can worsen when the effective support, branch count, rare-mode probabilities, proposal mismatch, or weight degeneracy worsens with dimension or horizon. Filtering and forward prediction both marginalize trajectories, but they are not the same integral run in reverse.
• Deterministic member propagation: roll each ensemble member forward without within-member noise. The resulting bundle approximates between-model disagreement, not total predictive spread. PETS instead combines probabilistic ensemble members with trajectory sampling.

Uncertainty can compound when recursive prediction enters locally expanding or unstable dynamics. Each state error then perturbs the next input and can be amplified. Contractive dynamics can instead damp errors, and stable stochastic systems may approach a bounded stationary spread. The worked examples below deliberately use settings in which spread grows; they demonstrate a mechanism, not a universal horizon law.

Before the worked example, we can illustrate the mechanism in isolation with a simple linear system whose one-step noise is constant. The figure below shows percentile bands of sampled rollouts: the band is narrow at step one and widens steadily with horizon, even though nothing changes about the per-step noise.

Uncertainty that compounds is not only a problem to mitigate. Several model-based reinforcement-learning methods make planners consume predictive uncertainty:

• Risk-sensitive objectives optimize a risk functional, such as expected utility, an entropic risk, a variance penalty, or a tail statistic. Chance constraints instead bound an estimated failure probability. Their guarantees depend on the predictive distribution, estimator, and optimizer.
• Disagreement penalties discourage candidate plans on which ensemble members disagree. Disagreement is a proxy for model risk, so this heuristic must be validated rather than treated as a trust certificate.
• Trajectory sampling for planning, as in PETS, rolls candidate action sequences through sampled probabilistic dynamics. Sampling represents both uncertainty sources, but robustness also depends on the objective, particle count, model fidelity, and optimization.

Why use a risk functional or chance constraint rather than expectation alone? Two candidate return distributions can have the same expected value while assigning very different probability to a failure tail. An optimizer that sees only expected return treats them as tied; a tail-sensitive objective or explicit failure-probability constraint can distinguish them, provided the predictive distribution and estimator are adequate.

For decisions that depend on branch identity, tails, or constraint violations, held-out one-step mean error is incomplete. A useful world model must represent the aspects of future uncertainty that affect the downstream objective and must be evaluated under that use.