Policies Learned in Imagination › Value Expansion and Synthetic Experience

Training a policy in imagination is one way to use a model. Another family uses it to improve value targets or augment replay data. Which approach performs better depends on model accuracy, data regime, task, and compute budget.

The distinction between the two families is a distinction in what the model is asked to do. In pure imagination, the model is asked to be an environment: it supplies entire trajectories, and the policy is trained on them. In value expansion and synthetic experience, the model is asked to be a data amplifier: it supplies a small amount of extra information that improves an otherwise model-free update. The model is still central, but its errors are more contained, because the update is anchored to real data rather than floating free in imagination.

An early ancestor is Sutton's Dyna architecture from 1990, elaborated in 1991. It interleaves updates from real transitions with planning updates from a learned model. The methods below refine how imagined transitions or returns enter a learner; they are not all interchangeable with arbitrary model-free updates.

Dyna separates two questions: where an update's transition comes from, and how the learner uses it. Real and simulated transitions can feed compatible updates, provided the algorithm handles their sampling distribution and model error. Generating extra experience is useful only when those conditions hold.

Model-based value expansion (MVE) uses short model rollouts to build value targets from real replay starts. For a state-value target, the model starts at a replay state and the current policy chooses the first and subsequent imagined actions. An action-value target can instead anchor its first action to the recorded replay action; later imagined actions still come from the current policy, not recorded future actions. The original formulation evaluates reward with the known task reward function; when that is unavailable, a learned reward head adds another error source. The following state-value equation depicts that learned-reward variant:

$$y_t^{H} = \sum_{i=0}^{H-1}\gamma^i \hat r_\psi(\hat s_{t+i}, a_{t+i}) + \gamma^{H} v_\xi(\hat s_{t+H}).$$

where:

• $y_t^{H}$: the $H$-step value expansion target at time $t$
• $\gamma$: the discount factor
• $\hat r_\psi(\hat s_{t+i}, a_{t+i})$: the model's predicted reward at the imaginary state $\hat s_{t+i}$ under action $a_{t+i}$
• $\hat s_{t+i}$: the $i$-th imaginary state along the rollout, initialized from a real state $s_t$
• $a_{t+i}$: an action sampled from the policy at the imagined state; in this state-value target, that includes $a_t$
• $\gamma^{H} v_\xi(\hat s_{t+H})$: the bootstrapped value term, weighted by the discount over the whole horizon

The parameter $H$ controls how many predicted rewards and transitions precede the bootstrap. A longer expansion downweights the terminal critic by $\gamma^H$ but exposes the target to more model error. STEVE mixes targets from multiple horizons, using inverse-variance weights estimated from ensembles of dynamics, reward, and value functions; those weights do not correct unestimated bias.

MVE can improve a value target when short-horizon model errors are smaller than the critic errors they replace. It can also make the target worse when the model is inaccurate. There is no universal ordering of model error, critic bias, and Monte Carlo variance. The useful horizon depends on those quantities and their covariance under the policy's state-action distribution.

Synthetic experience takes the Dyna idea literally. Rather than only changing the loss target, generated transitions are written into replay and sampled alongside real ones. The crucial design question is where those trajectories start. Model-based policy optimization (MBPO) branches from real replay states, takes a short number of model steps under the current policy, and adds those synthetic transitions to training. Its rollout length is tuned to model quality and training phase rather than fixed by a universal range.

Branching fixes the starting state to one observed in real replay. It limits the number of consecutive model steps but does not guarantee that subsequent policy actions or imagined states remain on the data distribution. MBPO's analysis makes the tradeoff between rollout length and model error explicit; the useful length depends on model quality and policy shift.

The containment strategy is to reduce, not eliminate, compounding error. Every synthetic transition is only a few model steps removed from a real start. Its distribution can still move far from the data if policy actions or one-step predictions do. Mixing in real data provides an anchor, but the mixture and resulting value estimates must be evaluated rather than assumed reliable.

Two further details matter: the synthetic-to-real sampling ratio, and which policy chooses imagined actions. Too much synthetic data can fit the critic to the model's distribution while concealing real-environment error in the training loss. Current-policy actions may improve relevance to the policy being evaluated but move rollouts away from the behavior data. This tension recurs in Offline and Conservative Model-Based RL.

Training loss on the synthetic mixture cannot establish accuracy under real rollouts. A held-out real transition set can test one-step value or model predictions; closed-loop environment rollouts test policy performance. Neither metric replaces the other. The action distribution deserves the same scrutiny because the model is used at states selected by the current policy, not merely at recorded replay states.

There is one more reason to prefer short rollouts, and it is the reason we return to in the next section: a long synthetic rollout gives the policy more opportunities to steer into a region where the model is wrong.