Policies Learned in Imagination › Summary

Learning policies in imagination converts a world model into a controller without planning at deployment time. The key ideas:

• Latent imagination rolls a learned transition under the policy without decoding every predicted observation. This can reduce compute, but the learned representation and reward head still need validation.
• Actor-critic learning inside the model can use $\lambda$-returns for critic targets and score-function or pathwise actor updates. The exact likelihood-ratio identity uses sampled return-to-go and discount weights; bootstrapped $\lambda$ advantages can be biased with an approximate critic. Pathwise updates use model derivatives and can be efficient on smooth, reliable models. Actor updates here do not optimize model parameters.
• Value expansion builds targets of the form $\sum_{i<H}\gamma^i \hat r_i + \gamma^H v_\xi(\hat s_H)$ by rolling the model forward from real states. The horizon trades model bias against reliance on the learned value function, whose weight decays as $\gamma^H$.
• Synthetic experience adds short model-generated branches from real replay states. Branching limits consecutive model steps but does not guarantee that the resulting distribution stays close to real data.
• Model exploitation remains a risk when a policy selects optimistic errors. The uniform infinite-horizon bound applies even to a model-selected policy but can be loose; average test error is not a substitute for checking policy-induced trajectories. The imagination gap can have either sign.
• Uncertainty penalties may help or hurt depending on calibration, reward scale, and task. In this toy run a stronger standardized disagreement penalty modestly improves real return. Missing coverage still motivates the targeted data collection developed in the next chapter.