Model-Based Reinforcement Learning › Limitations and Impact

The risk of model error is structural when predictive models are learned from finite data, even though a finite deterministic tabular model can become exact on each observed pair. Open-loop state error can remain bounded for contractive dynamics, grow linearly at $L=1$, or grow exponentially for $L>1$. Discounted value error has its own scaling and depends on whether error is measured in rewards, transitions, or Bellman backups. Common defenses include uncertainty estimates, short rollouts, pessimism, constraints, and validation on the decision distribution, but no one defense is universal. Model-based methods also trade data for computation. Under partial observation, the model needs a sufficient statistic such as a belief or recurrent state; one-step validation on aliased observations can otherwise look good while planning fails.

Model-based RL has produced influential sample-efficient systems. PILCO reported solving a physical cart-pole task with 17.5 seconds of interaction. PETS, MBPO, and Dreamer extended different model-use mechanisms to neural models and high-dimensional observations. DreamerV3 learned Minecraft diamond collection without human data by learning behavior from imagined rollouts while still collecting and acting on real environment experience. Dyna remains one important lineage among the decision-centric methods in Part VIII: Decision-Centric Research Lineages and the planning agents of Part VII: Planning and Agency.

For you as a practitioner, the chapter leaves you with a design checklist rather than a single recipe:

• Decide how the model is used, using planning, policy optimization, and imagined value learning as a practical taxonomy rather than an exhaustive partition.
• Keep imagined rollouts within the model's validated horizon. When the consumer uses a terminal value function, let that value carry the burden beyond the rollout.
• Use an uncertainty-aware diagnostic or objective when model uncertainty is material and can be estimated; an exact known model does not need a proxy for uncertainty.
• Evaluate the model on the decision distribution, not just held-out one-step accuracy, remembering the qualification test from Part I: The World-Model Idea.
• Report and monitor the real-interaction budget, the number that model-based RL exists to shrink.