Policies Learned in Imagination › Limitations and Impact

Learning a policy from imagined experience can move model computation out of the per-action control loop. Its value depends on the quality of the learned model, the policy update, and how performance is checked in the environment.

The Dreamer family is a clear example of latent imagination used to train behavior. MuZero and TD-MPC use learned models differently: they retain decision-time search or trajectory optimization. These lineages are evidence for several ways a model can improve control, not evidence that all successful model-based agents discard their planner. A reactive policy can be useful when a model is too costly to query on every control step, but training on many rollouts does not make it independent of model accuracy.

A policy trained from many starts may average some zero-mean sampling noise. It can also amplify a systematic optimistic error because the optimizer favors actions that exploit it. Online planners likewise evaluate many trajectories and often replan from new observations. The useful distinction is computational timing and the policy class learned, not an automatic robustness advantage for either family.

The central limitation is objective mismatch: the actor is trained under the model but judged in the environment. The uniform bound above covers even a model-selected policy, yet can be too loose to certify it. More useful warnings include:

• Imagined return improves while matched real-environment return stagnates or worsens.
• Actions or visited states depart from the model's training distribution.
• Multi-step prediction or reward error grows on states induced by the current policy.

These warnings are neither necessary nor sufficient. Measuring real return requires closed-loop environment rollouts; a held-out list of starting states alone cannot supply it.

An optimistic reward head can make an overshoot look cheap, as in the toy probe. Other models may be pessimistic or may fail through inaccurate transitions rather than large actions. Inspecting actual states, actions, and errors is more informative than inferring a mechanism from an imagined-return curve alone.

Each remedy changes a different part of the problem:

• Short horizons limit compounding steps but can lose delayed rewards; the observed sweep is nonmonotone.
• Ensemble penalties discourage estimated uncertainty but can miss shared model error; the observed penalty sweep improves return modestly for one seed.
• Behavior penalties discourage unsupported actions but neither guarantee safety nor impose a fixed performance ceiling.
• New real interaction can expose errors on visited states, at an environment-sample cost, when online data collection is allowed.

None of these interventions supplies a universal safety bound by itself. A finite penalty may be overcome by reward scale; a short rollout may still cross a bad one-step error; new data may not repair model misspecification. They are controls to test against the intended task, not certificates.

The imagination gap reflects both the model's error along a policy's trajectory and the policy found by optimization. Stronger optimization can discover optimistic mistakes, but it can also find actions that work better in the environment; the gap need not grow monotonically with iteration count, network size, or entropy changes. Model quality and optimizer settings must be evaluated together.

It is possible for a better solution to the imagined objective to perform worse in reality. That possibility is enough to require real-environment evaluation; it does not make stronger optimization inherently harmful. Report matched imagined and real returns, the training budget, and the states and actions where they diverge.

The practical takeaways are unglamorous and reliable:

• Measure the gap.
• Evaluate in the closed loop, not open loop, because open-loop prediction accuracy and closed-loop decision usefulness are different properties, as the coverage experiment showed.
• Sweep rollout horizons rather than assume shorter is always better.
• Keep some real data flowing.
• Treat every regularizer as a constraint to verify empirically rather than assume it helps.

The worked example supports a narrower conclusion. Model A has sparser goal coverage and worse open-loop error on the chosen probe. With bounded actions and the stated seed, it overshoots more and has a modest optimistic return gap; across three seeds, results vary. The horizon sweep is nonmonotone, and the penalty sweep improves one run modestly. These observations show why model diagnostics, policy rollouts, and sensitivity checks must be reported together.