Recurrent State-Space Models › Limitations and Impact

The limitations discussed here concern inference and rollout quality, representation, and computation.

The first theme is inference and rollout quality. During acting, observations can update the posterior; beyond the latest observation, imagination uses the prior with historical context. Some difference between these regimes is unavoidable because they have different information. Approximation, poorly fitted dynamics, and distribution shift can add further error. Held-out raw KL helps diagnose the representation, while point forecasts and distribution-sensitive tests assess its consequences. Free nats, balancing, and overshooting address different parts of training; none certifies trustworthy imagination.

The second theme is representation. A diagonal Gaussian restricts the conditional latent density at fixed context to an independent, unimodal family in its chosen coordinates. It does not forbid correlated or multimodal decoded physical variables: nonlinear transformations and mixtures over histories can produce both. Factorized categorical latents change those conditional modeling choices without guaranteeing an adequate belief. A finite-precision recurrent state can also lose relevant history, although some processes admit finite sufficient statistics even for unbounded histories. Low current-observation reconstruction error alone will not reveal every loss of predictive information.

The third theme is computation. An ordinary nonlinear GRU processes a length-$T$ sequence through $T$ sequential recurrent updates. Transformer training can process positions in parallel within each layer when the whole sequence is available, though this does not mean constant total work or universally faster training. Practical cost depends on sequence length, model size, hardware, and latency needs. The next chapter, Transformers and Neural State-Space Architectures, compares attention with structured and selective state-space sequence layers. S4 and Mamba also summarize history but are not RSSM Bayesian filters, and their update structure differs materially from a GRU. The autoregressive token models that follow inherit the evaluation questions: what history is represented, how do predictions change without future observations, and where do they fail?

PlaNet demonstrated latent-space planning from images. Dreamer learned an actor and value model through latent imagination; DreamerV2 reported Atari results, and DreamerV3 extended the evaluated domains to tasks including Minecraft. DayDreamer applied world-model learning to physical robots, including a quadruped learning to stand and walk in one hour. These systems reuse Bayesian terms such as prior, posterior, and belief rather than originating them. Their lasting practical question is whether an internal predictive state remains useful when it must continue without the observations that trained or corrected it.