Model-Based Reinforcement Learning › What Model-Based RL Adds to the Decision Loop

Building on the MDP formalism from Part II: Inference, Decisions, and Control, the agent faces a Markov decision process with transition kernel $p(s_{t+1} \mid s_t, a_t)$ and reward function $r(s_t, a_t)$. A model-free decision method learns a policy $\pi_\theta$ or value function $Q_\theta$ from experience without querying an explicit $p$ or $r$ during decision learning. An implementation may still train predictive auxiliary tasks; the distinction is whether a transition-and-reward model supplies the policy, value, or planning update. Model-based methods insert that intermediate object:

$$\text{experience} \;\longrightarrow\; \underbrace{\hat{p}_\phi(s', r \mid s, a)}_{\text{learned world model}} \;\longrightarrow\; \underbrace{\{\pi, Q, \text{plan}\}}_{\text{decision machinery}}$$

Read the diagram left to right as a data flow, not a one-time pipeline: real experience refines the model, and the model feeds decision machinery whose behavior determines what data is collected next. A learned model is an explicit, reusable summary of experience. It can be queried for actions not taken in a recorded transition when its representation and data support such generalization, and it can generate additional optimization inputs. Those synthetic transitions are not new independent evidence. Model-free policies, values, representations, and replay data can also be reused; the practical distinction is that a model exposes a transition-and-reward query interface.

Chapters 3 and 4 of this part already showed you both ends of this pipeline: system identification taught you how to fit a dynamics model from data, and classical optimal control taught you how to act with a known model. Model-based RL couples these activities online. Adaptive and dual control also study this controller-model-data feedback; MBRL brings the same problem into reinforcement-learning systems with learned, often high-capacity models. As the controller changes, its state-action distribution can change, so validation must track the distribution on which the model will actually be used.

The components of a model-based RL system are:

• Dataset $\mathcal{D}$ of real transitions $(s_t, a_t, r_t, s_{t+1})$ collected by earlier behavior. Its distribution often shifts as the policy changes, although it can be stationary under a fixed behavior policy.
• World model $\hat{p}_\phi$ and reward model $\hat{r}_\psi$, fit to $\mathcal{D}$ by likelihood or regression. MBRL often augments ordinary system-identification losses with uncertainty models, multi-step objectives, online updates, or control-aware weighting.
• Decision machinery, one of the three mechanisms described below: a planner, a policy, or a value function. This is the consumer whose needs the model must ultimately serve.
• Data-collection strategy, which decides what real experience to gather next, often by exploring where the model is uncertain or where the current policy performs poorly. This component closes the loop between the agent and the dataset. Model uncertainty can support model-aware exploration, although model-based and model-free systems can also use the same behavior policy.

Here is the trap that separates people who have run MBRL experiments from people who have only read about them. You fit the model to minimize one-step prediction error on the data you collected, exactly as system identification recommends. The model gets impressively accurate. You then let the agent plan with it, and the agent behaves worse than a model-free baseline. What went wrong?

The first time you see this, it feels like a contradiction. The model is measurably good; the decisions built on it are measurably bad. The resolution is that "good" was measured under the wrong conditions.

The mismatch has two root causes. First, distribution shift: as the policy improves, it visits states the dataset never contained, and the model extrapolates there. A neural network fit on random-walk data has no idea what the world looks like along the precise, purposeful trajectories a trained policy would take, and nothing in its training loss asked it to. Second, compounding: planners use the model recursively, feeding its own predictions back in as inputs, so one-step errors accumulate over the rollout horizon. Even a model whose one-step error is negligible on-distribution can drift badly over a twenty-step rollout, because step twenty's input is nineteen of its own guesses. Both deserve their own treatment, and we return to them in detail below.

There is also a subtler conceptual point, foreshadowed in the world-model qualification test from Part I: The World-Model Idea: open-loop prediction quality and closed-loop decision usefulness are different properties. A model can pass every prediction benchmark you throw at it and still produce bad policies, because policies live in a different part of the distribution than prediction benchmarks do. A benchmark scores the model where the data was; a policy uses the model where the optimizer steered. We will develop evaluation criteria for this in Part XI: Evaluation and Understanding of the handbook; for now, hold onto the distinction as a design principle. Whenever you train a model inside a decision system, ask not only "how accurate is it?" but "accurate where, and under what distribution of queries?"