World Models, SimPLe, and PlaNet › Controller learning in dreams

Once latent dynamics are combined with models of the task's reward and termination or discount, they can provide a learned simulator for control. The next question is how to couple that simulator to a decision mechanism. The chapter contrasts three answers: World Models and SimPLe train controllers or policies, while PlaNet uses online planning instead of training a separate policy network.

The first answer, used by World Models, is to train a compact controller separately from the vision and memory modules. A VAE compresses each $64\times64$ frame into $z_t$: the reported latent dimension was 32 for CarRacing and 64 for the VizDoom dream-training experiment. Given $(z_t,a_t,h_t)$, an MDN-RNN predicts a distribution over $z_{t+1}$ and advances its recurrent state to $h_{t+1}$. The paper's main controller equation is an affine map of $[z_t,h_t]$, while Appendix A.3's Doom-specific controller description specifies $[z_t,c_t,h_t]$, including the LSTM cell state. The released VizDoom code uses those inputs through a weight matrix without an explicit learned bias, followed by tanh and a task-specific action interpretation (state construction; controller implementation). Concentrating most complexity in the vision and memory modules keeps the controller's CMA-ES search space small.

CMA-ES treats return as a black box: it probes controller parameters with a population of perturbations and updates a search distribution from the better candidates. That made it practical for the small controller, but it did not guarantee protection from model exploitation. The paper documents controllers exploiting imperfections in the learned VizDoom dream and tests increased sampling temperature in the MDN-RNN as a mitigation. Controller size and gradient-free optimization are capacity and search choices, not demonstrated safeguards.

The second answer, used by SimPLe, is to train a policy inside a learned observation-space simulator. SimPLe stands for "Simulated Policy Learning." Its pipeline alternates between collecting real Atari interaction, fitting the video-and-reward model, and updating PPO mostly from simulated interactions. Real frames train and refresh the model and provide restart states. Policy learning is dominated by simulated rollouts, although the reported implementation also applied PPO directly to the much smaller real-data stream.

SimPLe uses a video model whose simulated environment emits visual observations to PPO; unlike World Models' controller inputs and PlaNet's planner state, its policy does not consume the model's internal latent state. Because the model predicts pixels directly, the policy can be trained on the same kinds of inputs a real agent would see, so the transfer from imagination to reality is at least in principle direct. Predicting a high-dimensional per-pixel distribution is computationally expensive; when the training objective aggregates similarly weighted per-pixel losses, large visual regions can dominate the gradient. Its clipped visual loss suppresses gradients from already-confident pixels, while a discrete stochastic latent can represent sharp conditional alternatives instead of averaging them into blur; neither removes the cost of pixel-space prediction. Multi-step rollouts can still accumulate visual artifacts and drift, so the training loop keeps imagined horizons short and re-trains the world model in each of its 15 data-collection iterations, after each 6,400-interaction batch.

PlaNet takes a different route: it trains no separate policy network and plans online in latent space with model-predictive control. Dreamer, in the next chapter, returns to policy learning and trains actor and value networks in imagination.

What unifies the three is that decisions are mediated by a learned predictive model, so decision quality can be affected by errors in the predictions or learned features that the policy or planner actually uses. SimPLe trains its policy in the learned simulator. World Models did so for VizDoom, whereas its CarRacing controller was optimized through real-environment rollouts while consuming learned VAE and RNN features. PlaNet trains no separate policy network and instead searches candidate actions online. Shorter rollouts and repeated grounding in real observations can limit open-loop drift, but they do not make transition or reward rankings correct. World Models additionally tested model temperature against dream exploitation. None of these devices is a guarantee.

These choices have different benefits and limitations; they need not be opposed. SimPLe, for example, combined short simulated rollouts with restarts from real buffered states. Shorter horizons can make the learning or planning target myopic. Restricting the controller to a smaller function class can reduce expressiveness, although the smaller search space may be easier to optimize. Restarting finite-horizon simulated rollouts from real buffered states limits each trajectory to the chosen number of model steps from a data state; it does not restrict the model itself. The three papers choose different points in this design space.

Imagined rollouts differ from real rollout data because their transitions come from a fitted model rather than the environment. Errors can be correlated along the rollout: if the model under-renders a moving sprite, that error can affect later inputs and predictions. SimPLe used a default simulated-rollout horizon of 50 model steps, restarting from four-frame observation stacks sampled from its real-data buffer; it ablated horizons of 25, 50, and 100 steps and ran the outer training loop for 15 iterations (Kaiser et al., 2019). A model can therefore fail locally in a region the improving policy increasingly visits even when its average prediction error looks acceptable.

This policy–model feedback loop is one mechanism behind failures of learning in dreams. As a policy changes, its state-occupancy distribution can also change. It may remain in well-modeled regions, generalize into sparsely covered regions, or seek states where prediction errors make return look spuriously high. More frequent visits to affected states increase their weight in on-policy model-error averages, all else equal, and create more opportunities for those errors to influence decisions; the resulting control harm need not grow monotonically. If the model is not refreshed, the policy may continue exploiting the same unsupported region while the error there remains untested. Periodic refresh with recent real data can interrupt or reduce this feedback loop by exposing the model to states the current policy visits, but finite data, capacity, and optimization mean that accurate coverage is not guaranteed.