Recurrent State-Space Models › Multi-Step Latent Rollout

Once trained, the model is used by imagining: rolling the latent state forward under actions, without observations. This section explains how rollouts work, when their error can grow, and what keeps them useful.

A rollout has two phases.

Burn-in. Run posterior updates over an observed prefix to initialize an approximate filtered belief. A known initial environment state can instead inform a suitable initializer; an unconditional constant is not a substitute for arbitrary known states. How much history is needed depends on observability and the model. To represent uncertainty over the installed belief, run several independently sampled posterior histories through the same observations and actions. Repeating one sampled endpoint would condition away the differences between those histories.

Imagination. From the installed belief, iterate the dynamics with the prior only, conditioned on a candidate action sequence:

$$h_{t+k} = f_\theta(h_{t+k-1}, z_{t+k-1}, a_{t+k-1}), \qquad z_{t+k} \sim p_\theta(z_{t+k} \mid h_{t+k})$$

where:

• $k$: the number of steps beyond the last observed time $t$; the pair of updates is iterated for $k = 1, 2, \ldots, H$ over the imagination horizon $H$
• $a_{t+k-1}$: the candidate action supplied by the planner at step $t+k-1$
• $p_\theta(z_{t+k} \mid h_{t+k})$: the prior, now the only source of the stochastic state because no observation arrives to correct it

The two updates play the same roles as during training: the GRU refreshes the memory from the previous belief and action, and the prior samples a fresh uncertain state. The agent then decodes whatever it needs from $(h_{t+k}, z_{t+k})$ — typically only the reward inside the planning loop, with observations decoded for inspection or evaluation rather than per candidate plan.

Two properties make latent imagination useful in PlaNet and Dreamer. Their action evaluation or actor training can operate through latent reward and value predictions without decoding images at every step. This avoids the image decoder's computation in those implementations. PlaNet evaluates candidate action sequences with a planner; Dreamer instead learns an actor through imagined trajectories and value bootstrapping. A single sampled rollout is one trajectory. Repeated independent posterior and prior noise draws approximate the model's conditional distribution of futures, which still needs validation. We will build the decision-making machinery in Part VII: Planning and Agency.

In System Identification, you saw why evaluating a one-step model recursively is a different test from predicting with fresh observations. An imperfect prediction can become an input to the next transition and affect later errors. Stable dynamics may attenuate that error; unstable dynamics may amplify it. Neither error growth nor monotonic widening is guaranteed merely by rolling a model forward.

Two contributions are useful to distinguish when interpreting forecast error:

• True conditional variability. Even a perfect conditional-mean predictor makes errors against individual stochastic futures. Process noise and uncertainty left after filtering affect both point forecasts and sampled forecasts. For a stable linear system, predictive covariance can approach a finite stationary level rather than grow forever.
• Model and inference error. Misspecified dynamics, imperfect optimization, a poor initial belief, or evaluation outside the training distribution can add error. Uncertainty about fitted parameters is one part of this problem, often called epistemic uncertainty; it is not a synonym for every approximation error. Recursion may amplify or attenuate these defects.

Accurately represented environmental variability should not be removed just to make intervals narrower. An inaccurate model may instead need better data coverage, inference, or dynamics fitting; shorter rollouts reduce reliance on extrapolation without necessarily fixing it. A point-output model has no interval, but that absence is not a declaration of confidence. Conversely, drawing a broad stochastic ensemble does not establish that the model knows its errors.

Use matched reference predictions when the environment is known. Below we implement the exact linear-Gaussian filter and its open-loop conditional mean and variance under the same observed prefix and actions. This gives a reference for irreducible squared error rather than comparing a random forecast sample with a conditional mean. We also measure interval coverage and width. Empirical 95% coverage at one horizon is only a marginal diagnostic: an overly wide interval can buy coverage, biases can be hidden across subgroups, and finite validation sets give noisy proportions. Neither coverage nor an RMSE curve alone decomposes model error from process noise.

Useful tools for managing rollout degradation include:

• Short horizons. PlaNet's reported setup plans 12 steps; the original Dreamer imagines 15 steps and bootstraps values beyond that window. PlaNet replans as observations arrive, while Dreamer acts through a learned policy conditioned on an updated latent state.
• Free nats and KL balancing. These adjust information regularization and gradient allocation. Free nats explicitly stop KL pressure below a threshold; neither technique guarantees a smaller prediction gap.
• Latent overshooting. PlaNet compares multistep prior predictions with posteriors inferred from real observations, adding direct multistep consistency pressure beyond one-step training.
• Stochastic latents. Sampling can retain modeled alternative futures instead of feeding back an intermodal mean. A sample lies in the learned prior's support, not necessarily in a well-trained region or in the true environment's support.
• Limited planner pressure. Optimizing strongly against an imperfect model can exploit its errors. Shorter horizons and uncertainty-sensitive objectives are possible defenses, not guarantees; Part XII: Reliable World Models develops this issue.

Evaluate the operating regimes separately. Posterior-pass error and prior-only rollout error depend on several components, but they ask different questions about access to observations. Reporting only posterior reconstruction can conceal open-loop weaknesses because each fresh observation offers a chance to correct the model. A point-error curve alone cannot assess distribution quality.