Policies Learned in Imagination › Policies Learned in Imagination

The planning methods in Part VII: Planning and Agency so far have used the world model at decision time. Sampling-based planning evaluates candidate action sequences, differentiable planning optimizes them through model gradients, and search expands imagined futures before committing to an action. Their model calls sit on the decision-time critical path.

That arrangement is demanding. A planner sits on the critical path of every decision, so it must fit within the control loop's latency budget. At 50 Hz, the entire sensing-to-action cycle lasts twenty milliseconds; state estimation, communication, inference, and actuation leave only part of that period for planning. The model also has to be accurate in a very specific way: not merely good on average, but trustworthy along the candidate trajectories the planner considers. Constraints may rule out large regions of state space, but a badly calibrated reachable candidate can still make a poor plan look attractive. In the language of classical control, this is the model predictive control regime, where an explicit model is queried online, repeatedly, and where closed-loop quality depends on the model along the plans actually considered and executed.

This chapter takes a different route: learn a policy inside the model, then act with that policy rather than running a planner for each decision. The agent rolls the model forward from states grounded in experience and uses synthetic trajectories to train a policy network. At action-selection time a deployed policy can map its current state estimate to an action without a planning rollout. Online variants may still collect data and update the model and policy between decisions.

The shift is a shift in when the model is used, and that single change ripples through everything else. At training time we can afford to run the model millions of times, to backpropagate through it, to compare it against real data, and to revise it. At deployment time we run a small network once per step. The model has become a teacher rather than an oracle, and the expensive, fragile, high-variance computation has been moved off the critical path entirely.

This is the amortization bargain: spend model compute during training to obtain a reactive controller with fixed per-step network inference cost. Dreamer illustrates this route; MuZero and TD-MPC instead retain learned-model planning at decision time. We will compare those distinct designs in Part VIII: Decision-Centric Research Lineages. First we need the mechanics of policy learning in imagination.

There is a classical analogy. In linear-quadratic control, a known or identified linear system and a specified quadratic objective yield an offline Riccati solution and a cheap feedback law. Here a learned transition and reward predictor supply imagined trajectories, while gradient descent fits a nonlinear policy. The task reward need not itself be learned as a preference; its predictor is learned because the actor trains without querying the environment on each imagined step. The analogy concerns moving computation out of the action-selection loop, not identical assumptions or guarantees.

This definition fixes the trajectory distribution over which the policy gradient is averaged: the one produced by the policy and learned model. A rollout may resemble real experience on the training distribution, but its transition law is still the model's. That difference underlies the failure modes we will study.

When real interaction is costly, model rollouts can provide additional training signal without another environment step, although those rollouts still consume compute and memory. The hazard is that the policy trains against a guess about the world. It can exploit optimistic errors that are reachable under its policy class and discoverable by its optimizer. A policy that scores well in imagination and fails in reality can therefore be optimizing its stated training objective faithfully.

The objective $\hat J(\omega)$ below is defined under the learned model. If a reachable model error makes a poor action look attractive, optimization may favor it over an action that works in the environment. The optimizer can be doing its job while optimizing the wrong objective.

We will build latent rollouts and the actor-critic losses that consume them, then examine value expansion, synthetic replay data, and ways to limit model exploitation.