Building a First World Model › From One-Step Predictions to Rolled-Out Futures

This section covers how to evaluate a world model's predictions, and why the evaluation protocol matters as much as the model itself.

There are two ways to ask a model about the future, and they produce very different answers.

Teacher-forced (one-step) prediction always conditions on ground truth. At every step, we filter the latent with the real observation, apply one prior transition, and score the prediction. Errors never accumulate because reality resets the latent each step. This measures the quality of a single transition.

Recursive (open-loop) rollout conditions on the model's own outputs. We filter once at the start, then chain prior transitions indefinitely, decoding observations along the way. Each prediction becomes the input to the next. Whatever error enters at step one is amplified and distorted by every subsequent step.

The names come from language modeling, where "teacher forcing" refers to training and evaluating a sequence model while feeding it ground-truth tokens rather than its own generations. The analogy is exact. A language model evaluated one token at a time with correct context can look fluent, while free-running generation drifts into repetition or incoherence; the mechanism of failure, consuming your own imperfect outputs as inputs, is identical. Keeping this analogy in mind is useful because the remedies also transfer: scheduled sampling, exposure to model-generated contexts during training, and explicit multi-step objectives all appear in both fields.

The gap between these two curves is arguably the single most important diagnostic in world-model research. A model can look excellent under teacher forcing and fall apart within twenty imagined steps. Since planning always involves recursive rollouts, teacher-forced accuracy alone tells you surprisingly little about whether the model is useful for decisions. We will measure both, and you should internalize the habit of distrusting one-step metrics.

To make the distrust concrete: imagine reading a paper reporting that a world model achieves 0.01 mean squared error on next-frame prediction. That number tells you the transition function is locally accurate. It says nothing about whether a fifty-step rollout stays in the vicinity of reality, and it says even less about whether returns computed over such rollouts rank action sequences correctly. A planner does not need the model to be right everywhere; it needs the model's relative ordering of candidate futures to survive contact with reality. These are related but distinct demands, and conflating them leads to models optimized for the wrong property.

Why does recursive error grow? Three mechanisms stack on top of each other:

• Error propagation: a slightly wrong latent state is fed back in, producing a more wrong latent state. For smooth systems the error often grows roughly geometrically at first.
• Distribution shift: the model was trained on latents produced by filtering real observations. During a recursive rollout it visits latents produced by its own predictions, a region of latent space it has rarely seen. Its behavior there is untested.
• Exploitation: when a planner searches over action sequences, it gravitates toward whichever regions of the model are optimistically wrong. We return to this failure mode shortly; it deserves its own treatment in Part XII: Reliable World Models.

It is worth separating the first two mechanisms carefully, because they demand different remedies. Error propagation is a property of the dynamical system interacting with an imperfect transition function: even a perfect model of a chaotic system diverges from reality given an imperfect initial condition, so some growth is unavoidable and the remedy is frequent correction, which is precisely what receding-horizon control provides. Distribution shift, by contrast, is a property of the training procedure: the model has simply never been asked to predict from self-generated latents, so its behavior there reflects extrapolation rather than fitting. That failure is addressable at the source, by training on rollouts, mixing imagined and real contexts, or regularizing the latent space so that self-generated latents stay near filtered ones. Confusing the two mechanisms leads to wasted effort, for example by building ensembles to fight a problem that was really a training-distribution problem.

None of these mechanisms is a bug in our implementation. They are structural properties of learning dynamics models from finite data, and every serious world-model system contains machinery designed specifically to contain them: multi-step training objectives, ensembles, uncertainty penalties, and frequent replanning.

One more caution before we build. Our model is trained to reconstruct observations and predict rewards, and it is tempting to equate low reconstruction error with a good world model. Resist that temptation. Reconstruction measures how well the latent supports describing the past and present. Decision-making needs the latent to support predicting futures under interventions. These overlap but are not identical. A latent could achieve beautiful reconstructions while discarding exactly the dynamical information a planner needs, or vice versa. The reward head partially bridges the gap by tying the representation to task-relevant structure, and Chapter 3 develops a proper qualification test for world models that goes well beyond reconstruction scores.

A concrete thought experiment makes the distinction vivid. Suppose the encoder learned to represent position with exquisite precision but packed velocity into a single noisy dimension, since velocity contributes less to immediate reconstruction than position does. One-step reconstruction scores would look excellent, yet every multi-step prediction would inherit the corrupted velocity estimate, and plans built on those predictions would misjudge momentum exactly as our feedforward baseline does below. Conversely, a latent that discarded fine visual detail irrelevant to dynamics, say the exact shade of the table surface, would reconstruct slightly worse but plan better. Which latent is the "better" world model? The question answers itself once you fix the purpose: model quality is not one number but several, covering predictive fidelity, representation adequacy, calibration of uncertainty, and usefulness for decisions, and different applications weight them differently. Holding these notions apart, rather than collapsing them into a single score, is a discipline we will maintain throughout the handbook.