Model-Based Reinforcement Learning › Model-Based Reinforcement Learning

Imagine a robot arm that must learn to insert a peg into a hole. A model-free reinforcement learning agent treats the arm as a black box: it tries actions, observes outcomes, and slowly accumulates value estimates through trial and error. On physical hardware this is painful. Every bit of learning costs real time, real wear on the motors, and real risk of damage. If a policy requires a hundred thousand training interactions, and each interaction takes a second of wall-clock time and a little life off the gearboxes, the arithmetic of deployment becomes unforgiving. A model-based agent takes a different route: it first learns how the world responds to its actions, then uses that learned model to rehearse. It can imagine thousands of peg insertions in the privacy of its own simulator, extract what it needs, and only occasionally return to reality to check its work. The expensive resource, real interaction, is spent sparingly; the cheap resource, computation inside the learned model, is spent liberally.

This is the promise of model-based reinforcement learning (MBRL): use a learned model to obtain more decision-making signal from each real interaction. Whether that promise is realized depends on the task, the data, and how the model is consumed. Model errors do not enter every decision uniformly; optimization can favor trajectories where the errors look advantageous. This chapter follows both sides of that bargain: how dynamics models support planning, policy optimization, and value learning, and how model bias, compounding rollout error, and model exploitation constrain those uses.

Throughout, keep two questions separate, because conflating them causes most MBRL confusion:

• What does the model predict? States, observations, rewards, sometimes whole futures.
• How does the agent use those predictions? To plan at decision time, to train a policy, or to train a value function.

The first question is about the model itself: its inputs, outputs, accuracy, and uncertainty. The second is about the decision machinery wrapped around it: whether predictions are consumed one step at a time, unrolled open-loop, or differentiated through. A given model can be paired with any of these consumers, and the same model can be excellent for one use and poor for another. Keeping the questions separate lets you diagnose failures precisely: when an MBRL system underperforms, the fault may lie in the model, in the consumer, or, most often and most subtly, in the interaction between the two.