Building a First World Model › Limitations and Impact

It is equally important to be clear-eyed about what this minimal system cannot do, because each limitation defines a branch of the research tree we explore in the rest of this handbook.

The most immediate weakness is compounding rollout error, which we measured directly. Our model is trained exclusively on one-step predictions, so nothing in its objective encourages stability under the recursive self-feedback that planning requires. Modern systems address this with multi-step training objectives, latent overshooting, and scheduled sampling, topics taken up in Part V: World-Model Architectures. Relatedly, our model is deterministic. Real environments are often genuinely stochastic, and a point-estimate transition model collapses multiple possible futures into one average future, which can be a future that never occurs. Representing outcome distributions, rather than averages, is a central theme of probabilistic world models.

The second cluster of weaknesses concerns the interface between the model and the planner. CEM optimizes predicted return, and predicted return is computed by a model with errors. A search procedure will happily find action sequences that exploit pockets where the model is optimistically wrong, achieving magnificent imagined returns that evaporate on contact with reality. This model exploitation problem grows more severe as the planner gets stronger relative to the model, and managing it, through uncertainty estimation, ensembles, pessimism, and careful evaluation, occupies much of Part XII: Reliable World Models. Our frequent replanning is a crude but effective patch: reality audits the plan every single step. But an agent that must replan at every step pays a heavy computational tax, and scaling world models to long-horizon reasoning requires planners that can trust their imaginations for longer stretches.

Third, our treatment of partial observability is minimal. The GRU accumulates information, but it maintains a single point-estimate belief with no notion of confidence. When two distinct situations produce the same observation history ambiguity, our model silently commits to one interpretation. Principled belief representations, where the agent knows what it does not know, are the subject of Bayesian filtering and belief-state methods in Part II: Inference, Decisions, and Control. And finally, our entire pipeline assumes a fixed, stationary environment and a fixed task reward. Environments that change, tasks specified in language, and models that must be updated online raise questions that Part VI: Learning World Models at Scale and Part IX: Foundation and World-Action Models begin to answer.

Against these limitations stands a remarkable record of impact. The recipe demonstrated here, encode, recurrently filter, predict forward, decode, and plan in imagination, is the direct ancestor of a family of systems that learned to master visual control tasks from pixels alone, culminating in agents that imagine entire episodes of experience internally and learn behaviors almost entirely inside their own dreams. Beyond robotics and games, the same template now underlies video prediction models, model-based offline reinforcement learning, and the emerging field of foundation world models trained on internet-scale video. The vocabulary you exercised in this chapter, latent state, prior and posterior transitions, rollout, imagined return, receding-horizon planning, is the shared language of all of them.

It is worth noticing how each limitation doubles as a research program with a recognizable lineage. Compounding error motivated the shift from purely one-step objectives to multi-step and rollout-based training. Determinism motivated probabilistic and ensemble transition models. Model exploitation motivated uncertainty-aware planning and pessimism. Point-estimate beliefs motivated variational and Bayesian state-space formulations. And the stationarity assumption motivates continual and online adaptation. Read in this light, the history of world-model research is not a sequence of disconnected tricks but a systematic response to the specific gaps this chapter's minimal system exhibits. You have now seen the gaps firsthand, which means the solutions ahead have something concrete to solve.