Recurrent State-Space Models › Recurrent State-Space Models

Watch a soccer ball roll behind a sofa. You can often predict where it will reappear, provided nothing hidden changes its motion. That familiar inference suggests a concrete engineering problem: maintain an internal state that keeps evolving when observations stop. A robot cannot pause its predictions whenever a camera is occluded. It must project forward from the information it has, while retaining uncertainty about what happens out of sight. In this chapter we build a small model that filters noisy observations and then predicts without seeing the future. We will also check where that model falls short.

In Part II: Inference, Decisions, and Control you met the classical answer, the Bayesian filter. A Kalman filter represents a Gaussian belief by its mean and covariance. With known linear transition and observation models, a Gaussian initial belief, and independent Gaussian process and measurement noise, its updates give the exact state posterior and minimum-mean-squared-error state estimate. Outside those conditions, ordinary Kalman updates generally cease to be exact Bayesian inference, although optimal linear estimation and other tractable special cases remain possible. In Part III: Representing Agents and Worlds you saw learned latent representations. The recurrent state-space model (RSSM), used by PlaNet and the Dreamer family, combines learned recurrent memory with stochastic latent dynamics.

An RSSM keeps two components of internal state. A deterministic recurrent state $h_t$, carried by a GRU or LSTM, provides memory. Deterministic describes its update given its inputs; it does not mean that the model is certain about the world. A stochastic latent $z_t$ is sampled from a conditional distribution. During filtering, a posterior network uses the current observation to infer that distribution. A prior network predicts it without the current observation, using past information through $h_t$. A KL divergence encourages the prior to match the observation-conditioned posterior and regularizes how much that posterior can differ. During acting, the agent can still filter its current observation; only its imagined future lacks future observations and therefore uses the prior alone.

The filtering/imagination distinction is the central operating boundary in the chapter, not a train/test split. Both regimes can be evaluated on held-out data, and filtering also occurs during deployment. The inference model asks, "what latent states explain what I have just observed?" The dynamics model asks, "what latent states should I predict before looking?" Training encourages consistency between them, but it cannot make unavailable future information available. A good predictive prior can therefore differ from the posterior even when neither is defective.