World Models, SimPLe, and PlaNet › Video prediction for Atari

SimPLe's world model is a video-prediction model, and to understand why it is hard and what it teaches, we should look at the objective directly. Given past frames $x_{1:t}$, past actions, and candidate future actions, an action-conditioned predictor models

$$p(x_{t+1:T} \mid x_{1:t}, a_{1:T-1}).$$

The future action sequence $a_{t:T-1}$ matters because later frames depend on actions that have not yet occurred at time $t$. Accurate pixel prediction requires accounting for the pixel-level effects of motion, appearance, interactions, occlusions, and actions that affect the target frames; useful control may depend on only a subset of those effects. Some factors are visually dominant but irrelevant to the current decision; others are small but useful. An on-screen score can be a useful visual cue, for example, but the environment still supplies reward as a separate variable. The pixels and reward should not be conflated.

The mismatch between what pixel loss rewards and what control needs is a central tension in video-model world models. Pixel loss is an imperfect proxy for decision-relevant modeling: optimizing it can favor visually dominant but control-irrelevant features. A model that reproduces the background perfectly but loses the agent, hazards, and their motion can achieve low pixel error while being useless for control. Conversely, a model with crude rendering that preserves the controllable objects and their interactions can be more useful for control than its pixel error suggests. SimPLe's primary policy evaluation reported game score and sample efficiency (Kaiser et al., 2019). Those test the learned policy's behavior, whereas pixel reconstruction error tests a different property of the model.

SimPLe's selected stochastic-discrete predictor was conditioned on four stacked frames and an action. It combined a convolutional video model with recurrent internal state across predicted steps (released configuration; video-model implementation). A separate auxiliary autoregressive LSTM generated its discrete binary latent bits within each step. The model downscaled Atari frames to $105\times80$ and used a 256-way categorical distribution for each output pixel channel. That per-pixel categorical likelihood is separate from the stochastic latent bits. It changes pixel regression into classification. Independent per-pixel categoricals can be multimodal at individual pixels, but their factorized joint cannot capture cross-pixel correlations needed to coordinate coherent alternative frames; the shared stochastic latent provides a mechanism for that coordination (Kaiser et al., 2019).

There are design decisions worth naming because they recur in later video-model world models. The first is input conditioning. An action-conditioned predictor needs the candidate action for each modeled transition; past actions can also help infer hidden state when the retained observations are insufficient. SimPLe feeds each next-frame prediction the action applied at that step, so its autoregressive rollout is conditioned step by step on the chosen action sequence. The second is stochasticity handling. The Atari emulator's core transition is deterministic given its complete state and applied action (Machado et al., 2018). ALE's optional sticky actions add stochasticity relative to the commanded action. Four observed frames need not reveal the complete emulator state. When the retained frame and action history aliases complete emulator states (for example, because relevant variables remain hidden, occluded, or affected by Atari's flicker), different futures may remain possible under the same candidate action, leaving conditional uncertainty for the model. SimPLe includes a discrete stochastic latent to represent conditional alternatives. The third is frame stacking and resolution. SimPLe worked at reduced resolution and stacked frames so the model could see motion (Kaiser et al., 2019). The fourth is evaluation for decisions versus evaluation for pixels. Pixel-averaged reconstruction error can look favorable when a model reproduces a large static background but misses a small control-critical object. The paper also reported model and reward-prediction diagnostics. Neither those diagnostics nor reconstruction quality alone establishes control utility.

A useful way to reason about video prediction is to keep three quantities separate. Let $c_t=(x_{1:t},a_{1:t})$ denote the available context for a one-step prediction.

• Irreducible conditional uncertainty under log loss is the entropy $H[p(\cdot\mid c_t)]=\mathbb{E}_{p}[-\log p(x_{t+1}\mid c_t)]$.
• Fitted-model discrepancy can be defined as $D_{\mathrm{KL}}(p(\cdot\mid c_t)\,\|\,\hat p_\theta(\cdot\mid c_t))$, averaged over contexts. The model's cross-entropy is their sum: $H(p,\hat p_\theta)=H(p)+D_{\mathrm{KL}}(p\,\|\,\hat p_\theta)$. Representative data, adequate capacity, and successful optimization can reduce the second term; more data alone does not guarantee that it shrinks.
• Cumulative rollout error is a chosen trajectory metric, for example $E_T=\sum_{j=1}^{T}\mathbb{E}[\|x_{t+j}-\hat x_{t+j}\|]$ under a specified coupling of true and model trajectories. Under a fixed action sequence, if the transition is uniformly $L$-contractive with $L<1$ in the chosen metric, true and model randomness are coupled consistently, and one-step discrepancy is at most $\epsilon$, then the per-step error is bounded by $L^j e_0+\epsilon(1-L^j)/(1-L)$; this prevents exponential amplification and gives a uniform per-step bound. The partial sums $E_T$ remain uniformly bounded as $T\to\infty$ only when the nonnegative per-step errors are summable; persistent error can instead make $E_T$ grow linearly. There is no universal theorem that it must grow superlinearly.

Separating these quantities points to different interventions. A richer stochastic predictor can represent conditional variation, but the distribution still has to be checked for calibration. Better coverage and modeling can reduce fitted-model discrepancy on the contexts represented in the data. Shorter rollouts and restarts from real buffered observation stacks reduce how long prediction errors feed back as inputs. SimPLe uses all three ideas, but the existence of a stochastic latent does not prove that its uncertainty is calibrated.

SimPLe limits compounding error by restarting simulated trajectories from real buffered frames after a finite horizon. MBPO starts short model rollouts from real replay states. Dreamer starts imagined trajectories from posterior states inferred from experience. These are related ways to limit reliance on unsupported long rollouts, not evidence of a direct lineage from SimPLe.

SimPLe's procedure alternates real-data collection, model fitting, and policy updates dominated by experience from the learned simulator; it also applies a much smaller set of PPO updates from collected real trajectories. This shared loop does not imply a one-way architectural lineage to the later methods discussed above. Its evidence remains specifically about data-efficient learning on Atari with an observation-space video model.