Model-Based Reinforcement Learning › Three Ways to Put a Model to Work

A useful taxonomy groups model use into three mechanisms: decision-time planning, policy optimization through imagined trajectories, and value learning from imagined experience. Many systems combine them, and some auxiliary-model or representation-learning methods do not fit the categories cleanly.

The most direct use: at every decision point, optimize an action sequence through the model, execute a prefix, and repeat. This is model predictive control (MPC), which you met in classical form in the optimal control chapter; the only change is that the dynamics come from $\hat{p}_\phi$ instead of a hand-derived ODE:

$$a_{t:t+H-1}^* \;=\; \arg\max_{a_{t},\dots,a_{t+H-1}} \; \mathbb{E}_{\hat{p}_\phi}\!\left[\sum_{k=0}^{H-1} \gamma^k \, \hat{r}_\psi(s_{t+k}, a_{t+k})\right], \qquad s_{t+k+1} \sim \hat{p}_\phi(\cdot \mid s_{t+k}, a_{t+k}).$$

This objective is risk-neutral: it scores a candidate by expected return under the model. Risk-sensitive or pessimistic MPC replaces that expectation with another functional. After executing $a_t^*$, the agent observes the next state and replans. This feedback usually limits exposure to open-loop drift, but it does not guarantee robustness to a biased model or make the inner optimization easy.

For continuous actions, sampling-based optimizers such as random shooting, CEM, and MPPI are common, alongside gradient methods and trajectory optimizers such as iLQR. Stochastic rollouts can still provide pathwise gradients through reparameterization or score-function estimates; sampling is attractive when such gradients are unavailable, unstable, or not worth their cost. PETS (Chua et al., 2018) paired CEM with a probabilistic neural-network ensemble and reported strong sample efficiency on its continuous-control benchmarks. Tree search is important for many discrete-action problems; MuZero searches in a learned latent model, as discussed in Part VIII: Decision-Centric Research Lineages.

Planning has a distinctive profile:

• No separate policy training is required. Plain online MPC selects actions directly, although learned proposal policies and planner warm starts are common. With learned nonlinear dynamics, the inner optimization is often nonconvex; linear MPC with convex costs and constraints can instead be convex. Numerical behavior and closed-loop stability depend on the full formulation and solver.
• Decision-time compute. Sampling budgets range widely with the optimizer, horizon, model, and hardware. This can be difficult under hard real-time constraints, although learned-model MPC can run at high rates in suitably engineered systems.
• Short horizons are natural. Because reality corrects the state estimate after each executed action, MPC can succeed with horizons where open-loop planning would fail. The model is exposed to at most $H$ consecutive predicted transitions even when the task lasts much longer.

The second mechanism treats the model as a differentiable simulator and backpropagates value through it to train a policy directly. For the expression below, assume a deterministic differentiable transition model, a deterministic differentiable policy $a_k=\pi_\theta(s_k)$, and a differentiable modeled reward. Then $\hat{s}_{t+k+1} = f_\phi(\hat{s}_{t+k}, \pi_\theta(\hat{s}_{t+k}))$, the finite-horizon return is differentiable in $\theta$, and you can compute gradients through the imagined trajectory. A stochastic policy instead needs reparameterization or another valid gradient estimator.

$$\frac{dJ}{d\theta} = \sum_{k=0}^{H-1}\gamma^k\left[ \frac{\partial \hat r_k}{\partial s_k}\frac{d s_k}{d\theta} + \frac{\partial \hat r_k}{\partial a_k} \left(\frac{\partial \pi_\theta}{\partial\theta} + \frac{\partial \pi_\theta}{\partial s_k}\frac{d s_k}{d\theta}\right) \right],$$

where $a_k=\pi_\theta(s_k)$ and the state sensitivity obeys the corresponding recursion through $f_\phi$. These sensitivities multiply repeated model Jacobians, so they can vanish or explode with horizon. PILCO (Deisenroth and Rasmussen, 2011) used Gaussian-process dynamics, approximate analytic moment propagation, and analytic policy gradients, solving its reported control tasks with very little interaction. Its posterior uncertainty is meaningful under its modeling assumptions, but is not automatically calibrated. Gradient-based and sampling-based optimizers can both exploit model error; the result depends on their search pressure, regularization, and the geometry of the learned model.

The Dreamer family learns latent dynamics and trains actor and value functions from imagined latent rollouts. The original Dreamer propagates actor gradients through imagined dynamics; estimator details vary across versions. A latent state can omit task-irrelevant observation detail, but it is not necessarily lower-dimensional and does not shorten the temporal chain by itself. Part III: Representing Agents and Worlds develops the representation machinery.

The third mechanism is the oldest and, in some ways, the most durable: use the model to generate synthetic transitions, and feed them to an otherwise ordinary model-free learner. The value backup

$$Q(s,a) \leftarrow Q(s,a) + \alpha\left[r + \gamma\max_{a'}Q(s',a') - Q(s,a)\right]$$

has the same algebra for a real or modeled transition. In practice, model bias and the synthetic-data distribution often require different rollout lengths, sampling ratios, weighting, or pessimism. The model supplies additional optimization inputs and can branch from real states into continuations not present in the replay buffer; it does not supply additional independent evidence.

This is the Dyna family, which gets its own section below. MBPO (Janner et al., 2019) is a modern relative that trains soft actor-critic on short model rollouts branched from real replay states and reported strong sample efficiency on its benchmarks.

A useful refinement is model-based value expansion (Feinberg et al., 2018): unroll $H$ modeled steps from a real state, sum predicted rewards, and bootstrap from a terminal value estimate. Increasing $H$ reduces reliance on the terminal bootstrap, but it can increase sampling variance and exposure to accumulated model bias; the balance depends on the estimator, model, and task. The rollout rewards are model predictions, not additional real reward observations.

The three mechanisms are compared in Table model-uses:

These are not rivals so much as a menu. PETS combines a learned ensemble model with planning. Dreamer combines value learning with gradients through the model. MBPO combines Dyna-style imagined data with off-policy value learning. The table's failure column is a diagnostic aid: ask how the model is consumed, then ask how the method limits harmful distribution shift and model exploitation.