World Models, SimPLe, and PlaNet › Limitations and impact

The three papers taught different lessons, and each carries a limitation. World Models showed that a compact controller trained entirely in a learned VizDoom dream could transfer to the real VizDoom environment; its CarRacing controller was instead optimized in the real environment. The separately trained VAE and memory need not retain every task-relevant feature, the no-hidden-layer controller limits expressiveness, and the paper directly observed dream exploitation. SimPLe showed that an observation-space video model could support data-efficient Atari policy learning, but its pixel objective is not directly decision-aware, and errors can feed back even through its deliberately truncated multi-step rollouts. PlaNet used an RSSM for online latent planning on six image-based DeepMind Control Suite tasks. It also proposed latent overshooting as a training regularizer, though its reported final RSSM did not require it. Its CEM search adds execution cost and can still prefer action sequences whose predicted returns are wrong.

These limitations arise from the model objectives, the controller or planner design, and the way the two are coupled.

• World Models: The VAE and MDN-RNN remain active during deployment to supply $z_t$ and $h_t$; the paper's Doom-specific controller description and the released VizDoom controller also consume the LSTM cell state $c_t$. Their separate training objectives may omit task-relevant information, while a controller trained in the fixed dream can exploit its errors.
• SimPLe: The observation-space simulator must be accurate enough for policy transfer. Its pixel likelihood is not directly optimized for control. The clipped visual loss stops gradients from already-confident pixels; the authors conjectured that this reduces background dominance. A separate reward head adds a task-specific signal. Remaining observation and reward errors can still bias the learned policy.
• PlaNet: At each real decision step, the model is queried over candidate action sequences of its configured fixed decision-step horizon; the selected first action is repeated for a task-specific number of environment transitions (Algorithms 1–2). Re-observation limits open-loop commitment, but useful multi-step transition predictions and return rankings remain necessary.

Taken together, the three establish a spectrum of couplings between the world model and the controller, and they make one tension in model-based RL explicit. For these return-maximizing methods, expected return is the control objective; global predictive fidelity alone is not. Low average next-frame error does not by itself guarantee a good policy, and relatively high pixel error does not by itself rule one out, because control depends on errors in the decision-relevant quantities and trajectories queried by the particular controller. For the model's contribution to success, the relevant question is whether its predictions are decision-faithful on the states, action sequences, and horizons the controller queries, including the return rankings or value updates needed to choose good actions. Planner or policy quality, optimization, exploration, and data collection remain separate requirements. This framing connects several mechanisms discussed here: short grounded rollouts, KL-regularized latents, constrained policies, and model refresh.

The impact is clearest in the design questions that remained active. Dreamer retained an RSSM-like imagination model but learned actor and value networks instead of planning with CEM. It started imagined trajectories from posterior model states inferred from stored experience. DreamerV2 retained that starting point while replacing continuous Gaussian stochastic states with categorical ones. IRIS took another route, encoding frames as discrete observation tokens and predicting token dynamics with an autoregressive Transformer. These are distinct representation and control choices, not one direct lineage from SimPLe. Together, World Models, PlaNet, and SimPLe show observation-space simulators and learned latent-state models coexisting, with prediction coupled to recurrent controller features, simulator-trained policies, or online latent planning.