Recurrent State-Space Models › Hidden State: What Deterministic and Stochastic Each Buy You

This section builds the two ingredients separately before combining them. PlaNet's experiments motivate the combination, but neither ingredient is a universal requirement for a capable world model. We need to distinguish how a state is computed from what distribution its outputs can represent.

Recall from Bayesian Filtering and Belief States that an observation-action history contains the information available for inferring hidden state. Storing that raw history costs more as an episode grows. Some observations are redundant, while repeated noisy measurements can still help reduce uncertainty. A belief $b_t = p(s_t \mid o_{\le t}, a_{<t})$ is the distribution of hidden environment state $s_t$ given observations through $t$ and earlier actions. It is a sufficient summary for the usual known-model POMDP prediction and decision problem, but a general belief distribution need not have an exact finite-dimensional representation. Classical filters exploit tractable special families. A learned world model instead fits a compact representation intended to retain the information its prediction tasks need.

There are two computational choices to separate:

• A deterministic hidden vector is a fixed function of its inputs and parameters. It can store point estimates, probabilities, variances, or other learned features.
• A stochastic latent is a random variable. Its distribution represents modeled alternatives; one sample is only one realization, not the distribution itself.

A Kalman filter is a useful counterexample to the idea that deterministic memory means certainty: its mean and covariance are updated deterministically, yet jointly describe an uncertain state. Likewise, a neural vector can encode the weights, means, and scales of a mixture distribution. The RSSM chooses a different implementation: recurrent memory plus sampled latent states. That choice can help retain information and generate alternative trajectories, but it does not by itself guarantee good inference or calibrated uncertainty.

A simple learned filter is a recurrent neural network that ingests observations and actions and maintains a fixed-size vector:

$$h_t = f_\theta(h_{t-1}, a_{t-1}, o_t)$$

where:

• $h_t$: the learned hidden state at time $t$, a fixed-size vector that summarizes the observation and action history
• $f_\theta$: the learned recurrence, for instance a GRU or LSTM cell, with parameters $\theta$
• $a_{t-1}$: the action taken at the previous step
• $o_t$: the observation arriving at the current step

A related recurrent-memory pattern appears in Ha and Schmidhuber's World Models (2018). Its LSTM consumes VAE codes and actions and outputs a mixture distribution, not a point prediction. In our simpler recurrence, $h_t$ is a learned predictive summary, not a guaranteed sufficient statistic. A fixed-width representation puts pressure on the network to retain task-relevant information; it may also forget information that later turns out to matter. The prediction objective determines what is rewarded, and held-out tests tell us whether that training signal was enough.

Deterministic state has real strengths:

• Repeatable updates. With fixed inputs, initialization, parameters, and deterministic numerical execution, the same recurrence returns the same state. This removes latent-sampling variance from that update, not representation or floating-point error.
• Direct differentiation. Ordinary backpropagation follows the recurrence without a fresh sampling operation on each edge. Gradients can still vanish or explode.
• Simple inference. The point-output baseline below needs one recurrent pass and no variational posterior.

The limitation appears when we attach a single point output and train it with mean squared error. If the next observation has two plausible modes, that output cannot report both. Conditional squared risk decomposes into the irreducible conditional variance plus the squared distance from the conditional mean, so the MSE-optimal point forecast is the mean. For a balanced symmetric bimodal distribution, that mean lies between the modes. A forecast of a ball on the left or right of a pillar can therefore average into a low-probability location in the middle. The cause is the point-output objective, not deterministic recurrent memory: a distributional head can report both alternatives.

The failure is easiest to see with numbers rather than words, so we make it concrete before moving on.

The plot illustrates mode averaging: the conditional mean lies in the low-density valley between the modes, not at the global minimum of the density. Enlarging a model while retaining an MSE point-output objective does not change the optimum into two alternatives. Changing the output distribution or the prediction objective addresses a different problem than adding hidden-state capacity.

A point output also leaves uncertainty unreported. Consider two grasps with the same predicted mean outcome but different failure risks. A planner receiving only those mean outcomes cannot distinguish them by uncertainty. Risk-sensitive decisions or information gathering may need a distribution, quantiles, or other uncertainty-sensitive information. These can come from deterministic memory with a probabilistic head, from ensembles, or from stochastic latent rollouts. A point forecast can still be useful for decisions; it simply does not answer every decision question.

One way to generate alternatives is to make part of the latent state stochastic. Instead of reporting only a point, the model predicts a distribution and samples from it. One trajectory may go left and another right if its learned distribution represents both outcomes. Sampling represents what the model believes, not a certificate that the belief is correct. Calibration must be checked against held-out outcomes.

Stochastic latent sequence models precede RSSMs: variational RNNs (Chung et al., 2015), stochastic RNNs (Fraccaro et al., 2016), deep variational Bayes filters (Karl et al., 2017), and structured inference networks (Krishnan et al., 2017) develop related latent-state and inference ideas. In these models, distinguish the approximate latent distribution from a sample drawn from it. For an RSSM, the joint distribution of $(h_t,z_t)$ matters: $h_t$ varies across previously sampled latent histories, even though each recurrent update is deterministic conditional on its inputs.

Stochastic recurrence can make retaining information and optimizing long sequences difficult. PlaNet reported that its combined recurrent and stochastic state performed substantially better than the particular single-path variants it tested. That result does not establish a universal law that each additional sample increases gradient variance. For example, in $z_t=z_{t-1}+\theta+\epsilon_t$ with a terminal linear loss $z_T$, the pathwise derivative is exactly $T$ for every noise draw. Actual gradient variance depends on the transition, objective, and estimator. The RSSM's deterministic path offers a route for recurrent information without requiring a fresh sample on every edge.