Model-Based Reinforcement Learning › Summary

• Model-based RL inserts a dynamics and reward model between experience and decisions. When data and representation support the query, that model can evaluate alternative actions and generate synthetic optimization inputs.
• A useful taxonomy asks whether the model supports decision-time planning, policy optimization through imagination, or value learning from synthetic transitions; many algorithms combine these uses.
• Under the stated same-start, uniform-error, and Lipschitz assumptions, the worst-case open-loop state-error bound is finite, linear, or exponential according to transition sensitivity. Discounted performance bounds must likewise state the error metric and assumptions.
• Model exploitation occurs when optimization favors optimistic model errors. Ensembles, short branched rollouts, pessimism, constraints, and decision-distribution validation are common defenses, none of which certifies correctness alone.
• Dyna interleaves real updates, model learning, and backups from visited model entries. Direct planners and differentiable policy-search methods need not follow this loop.
• In the deterministic gridworld experiment, both Dyna variants reached a smoothed return of $-2$ in about one-third of Q-learning's mean real interactions across seeds. That result is specific to the implementation and hyperparameters shown.

The next part of the book turns to the representation question this chapter deliberately set aside: what should the model predict over? Raw observations are often an awkward target for control-relevant prediction, and the road to latent, memory-equipped, object-centric world models starts with a closer look at where observations come from.