World Models, SimPLe, and PlaNet › Latent planning with learned state-space models

Now we turn to PlaNet's online-planning approach, where its RSSM supports fast latent-space rollouts. Given a learned latent dynamics model, an inferred current latent state, and a reward predictor, we can evaluate candidate action sequences by imagining their latent trajectories and summing predicted rewards. We then execute the first planned action for the task's configured action-repeat interval, observe the resulting real frame, infer a new latent, and repeat. This is model-predictive control (MPC) in latent space, and it is the direct latent-space counterpart of the sampling-based MPC we discussed in Part VII, Chapter 1.

PlaNet's planner uses the cross-entropy method (CEM), a sampling-and-elite-refitting optimization procedure (Rubinstein, 1997). The following operational summary combines PlaNet's Algorithm 1 control loop and supplementary Algorithm 2 CEM planner with the released implementation's action clipping. At each time step:

1. Infer the current latent state from the observation history using the posterior.
2. Sample $N$ candidate action sequences of horizon $H$ from an initial Gaussian proposal, then clip each sampled action to the environment's valid range before evaluating it.
3. For each candidate, roll the latent forward using the prior transition (no observations available for future steps), accumulating predicted reward from the reward head.
4. Keep the best $K$ elites, refit the Gaussian proposal to their bounded action sequences, and repeat for a few iterations.
5. Select the first action vector of the final proposal mean; during data collection add the configured exploration noise, then apply the bounded action for the task's action-repeat interval, observe, and re-infer the latent.

Each of these steps has a rationale worth understanding. Posterior inference incorporates the latest real observation before planning. Sampling lets the planner search a non-convex return surface, while prior rollouts are necessary because future observations are unavailable. The elite refit concentrates the proposal on higher-scoring action sequences. Re-observing and replanning after each decision-level action, after its configured action-repeat interval, limits open-loop error accumulation, but it does not repair a transition or reward model that ranks the local candidates incorrectly.

One conditional planning objective, after fixing a sample $z_t$ from the current posterior, is:

$$\arg\max_{a_{t:t+H-1}} \; \mathbb{E}_{p_\theta(z_{t+1:t+H}\mid h_t,z_t,a_{t:t+H-1})} \left[ \sum_{j=1}^{H} r_\theta(h_{t+j},z_{t+j}) \right],$$

where:

• $a_{t:t+H-1}$: the candidate action sequence of horizon $H$ that the planner is choosing
• $H$: the planning horizon, the number of future steps the planner looks ahead
• $r_\theta(h_{t+j},z_{t+j})$: the predicted reward after action $a_{t+j-1}$ has advanced the model state
• $h_{t+j}=f_\theta(h_{t+j-1},z_{t+j-1},a_{t+j-1})$: the deterministic recurrent update
• $p_\theta(z_{t+1:t+H}\mid h_t,z_t,a_{t:t+H-1})$: the joint action-conditioned rollout distribution, equal to the product of the sequential priors $p_\theta(z_{t+j}\mid h_{t+j})$
• $\mathbb{E}[\cdot]$: expected return over that joint stochastic trajectory, conditional on the sampled current state and the full candidate action sequence

The objective aligns action and reward timing: every candidate action affects one included next-state reward, including the final action. Monte Carlo rollouts can estimate the expectation. PlaNet's published Algorithm 2 samples a current posterior state for each candidate trajectory, so its candidate-trajectory distribution includes current-state uncertainty as well as future transition uncertainty. It uses one sampled trajectory per candidate action sequence rather than averaging repeated trajectories for the same candidate; the CEM population supplies the broader sample set. The released agent instead obtains one posterior state per real decision, and the planner implementation tiles that state across its CEM candidates. Its future transition samples can still differ across candidates. This paper-versus-code distinction does not change the receding-horizon control loop, but it matters for which uncertainty is sampled within one planning call.

Two points matter in latent planning. The first is a risk shared by any planner using approximate dynamics; the second is specific to learned latent representations and inference.

First, the planner's return rankings depend in part on how the model extrapolates along candidate action sequences. If an action sequence drives the latent into a region with no training support and receives a spuriously high predicted return relative to alternatives, the planner may prefer it over better real-world sequences. This is model exploitation: short planning horizons and re-planning after every step limit exposure to extrapolation regions, though they cannot eliminate it.

Second, PlaNet plans in a learned latent trained jointly for observation reconstruction, reward prediction, and dynamics. This training may preserve decision-relevant variables, but it does not guarantee that they are emphasized over visual nuisance information. PlaNet does not render candidate future frames for planning. For accurate reward and transition predictions, the complete recurrent state $[h_t,z_t]$ must retain the information those models require, though it may also retain nuisance information. If that complete inferred state omits a task-relevant variable, more search over it cannot recover the missing information.

The contrast with MPC over an explicitly specified state-space model is instructive. Classical MPC can also face model mismatch, disturbances, state-estimation error, constraints, and optimization error (Rawlings et al., 2017). Learned latent MPC additionally introduces potential representation error, while approximation in its learned encoder or posterior is a source of state-estimation error. Replanning after each observation incorporates new evidence and limits commitment to an open-loop prediction; it does not guarantee recovery of the true state under partial observability or encoder error (PlaNet).

Putting the three side by side clarifies what each contributed and what each does at test time:

• World Models: A VAE encodes each frame, an MDN-RNN predicts compressed dynamics and updates memory, and a tiny controller acts on latent and recurrent features. The paper's main equation uses $[z_t,h_t]$ plus a bias; Appendix A.3's Doom-specific controller description and the released VizDoom state construction use $[z_t,c_t,h_t]$. The released controller has no explicit learned bias. CarRacing controller optimization used the real environment; the VizDoom controller was trained in the learned dream. At deployment, the VAE and MDN-RNN remain active because they supply the controller inputs.
• SimPLe: An observation-space video model predicts future frames and rewards from recent frames and actions, using a discrete stochastic latent internally. The policy is trained with PPO primarily on rollouts from the simulated environment; collected real-environment trajectories also supply a much smaller set of PPO updates. At execution, the policy consumes real frame observations and the video model is not needed.
• PlaNet: An RSSM with deterministic and stochastic state components, plus observation and reward heads. Online CEM searches action sequences by rolling the latent transition and reward models forward. At each real decision step, the encoder and posterior incorporate the new observation before replanning. Each chosen action is then applied for the task's configured action-repeat interval. The observation decoder is not used inside planning, and no separate policy or value network is trained.

These are three different couplings: learned compressed features and memory feeding an evolved controller; a refreshed observation-space simulator supplying PPO experience; and online action-sequence search scored by a latent model. Each coupling helps identify which model errors affect decisions on the states, actions, and horizons actually queried.

World Models uses its memory in two roles: to generate VizDoom dream transitions during controller training and to provide recurrent features to the deployed controller. The paper's general controller equation uses $h_t$, but Appendix A.3's Doom-specific controller description already includes both $c_t$ and $h_t$; the released VizDoom state construction follows that task-specific input choice. SimPLe needs video rollouts accurate enough that a policy trained in the simulator transfers to Atari. PlaNet needs sufficiently accurate multi-step return rankings across its planning horizon. Latent overshooting directly encourages multi-step latent-prior consistency against later posteriors and can support return ranking only indirectly. The released configuration defaulted its scale to zero. Receding-horizon execution limits how long PlaNet commits to a candidate sequence, but choosing the first action still depends on all rewards predicted across that sequence.