World Models, SimPLe, and PlaNet › World Models, SimPLe, and PlaNet

Three other model-based reinforcement learning (MBRL) methods provide a useful contrast: they fit dynamics in observation space or in low-dimensional hand-designed state spaces, then used those dynamics for planning or for generating synthetic rollouts. PILCO learned probabilistic dynamics over compact, low-dimensional continuous state vectors. PETS used probabilistic neural-network ensembles to plan action sequences. MBPO used model ensembles for short synthetic rollouts that augment a model-free learner. Those methods share a common feature: their learned models predict task-defined state-vector transitions used for planning, model-based policy optimization, or generating policy-training data, rather than dynamics in an image-compression latent state. When a low-dimensional state vector is sufficiently observed and predictive for the task, direct state-space modeling and planning can be practical. When the observation is a 210-by-160-pixel Atari screen or a first-person 3D view, per-pixel modeling and planning become computationally demanding and can scale poorly.

This chapter compares three responses, each with a different place for prediction. World Models compresses images and predicts dynamics in that compressed representation. SimPLe instead rolls an observation-space video model forward; its discrete stochastic latent is an internal source of variation, not the compact state consumed by the policy. PlaNet learns a compact recurrent state-space model and plans through it online. Treating all three as latent-dynamics methods would hide the most useful contrast.

First posted to arXiv within roughly one year and addressing overlapping image-based model-learning problems, the papers are historically adjacent but conceptually distinct. All three use a learned predictive model to support decisions. They differ in what is predicted, what the decision mechanism consumes, and whether a controller is trained or actions are planned online. Reading them together shows why the model's coupling to a policy, planner, or value function matters: that coupling helps identify which predictions matter, alongside the task and the states, actions, and horizons the decision procedure queries. Low average pixel error does not by itself guarantee a useful controller, and relatively high pixel error does not by itself rule one out, because control depends on errors in the decision-relevant quantities and trajectories queried by the particular decision mechanism. The Dreamer family, which we examine next, pushes this insight further. Here we build the foundation.

To keep the concepts clean as we proceed, it helps to hold four separate notions of "model quality" in mind, because the rest of this chapter repeatedly depends on the distinction. Predictive fidelity asks how accurately the model forecasts its declared prediction targets, such as future observations, latent transitions, rewards, or termination and discount signals. Representation quality asks how well the latent organizes the world into useful, disentangled, or stable factors. Uncertainty quality asks whether predictive uncertainty is calibrated to the conditional variation remaining given the model's information (from stochastic transitions, observation noise, or unresolved hidden state), and whether epistemic uncertainty reflects gaps in the model's knowledge. Those are related but distinct questions. Decision usefulness asks whether the model supports good actions once it is plugged into a planner, a policy, or a value function. These four can be related, but they need not move together. Optimizing one does not ensure improvement in another. The three papers emphasize different predictive targets, representation choices, and decision couplings; none of them, by itself, establishes calibrated predictive or epistemic uncertainty.

Our plan: first, we develop variational latent dynamics in general and then the recurrent state-space model (RSSM) used by PlaNet, which gives us a concrete reference point for later latent models. Next, we look at controllers learned in dreams, contrasting the World Models training objective with the SimPLe procedure. Then we examine video prediction for Atari as a modeling problem, since SimPLe's learned environment is a video-prediction model and many of its model-related difficulties arise there. Finally, we look at latent planning, in particular PlaNet's CEM-based planner over the RSSM latent state, and explain why planning in latent space is not the same as planning in the true state.