The Dreamer Family › Summary

The Dreamer family replaces per-step trajectory search with an actor and value function trained on latent imagination. Actor inference avoids a fresh online search. Published DreamerV3 experiments report one configuration across several benchmark suites, but additional imagination does not guarantee monotonic improvement or real-environment transfer.

The key ideas to take away:

• The RSSM combines recurrent memory with stochastic latent variables. It separates the prior used in imagination from the observation-conditioned posterior. Stochastic state can encode ambiguity but is not automatically calibrated uncertainty.
• Latent imagination rolls the model forward using the prior and current policy. V1 backpropagates through dynamics with reparameterized continuous actions or straight-through discrete actions; V2 selects between dynamics and REINFORCE estimators by domain, and V3 uses REINFORCE with stopped world-model gradients. The critic uses horizon-bootstrapped lambda-returns.
• DreamerV2 combines categorical latents, straight-through sampling, KL balancing, and domain-specific actor-gradient estimators. Its Atari result belongs to the complete system, not the latent change alone.
• DreamerV3 uses symlog vector encoder inputs and decoder targets, symexp-spaced two-hot reward and value predictions decoded as an original-scale weighted average, normalized actor advantages, free bits, uniform mixing, and further architecture and optimization choices; images retain their convolutional path. One reported configuration was evaluated across Atari, DMLab, control, and modified MineRL tasks.
• Limitations are real: compute comparisons are implementation-dependent, long-horizon sparse-reward exploration is not solved, imagination can diverge from reality, and semantic interpretability is not guaranteed.

The version history is qualitative rather than a set of commensurate capability scores:

The most important skill to develop from this chapter is the ability to distinguish imagined performance from real performance. Dreamer's benchmark results show that latent imagination can train strong policies, while the toy experiment shows why transfer must be measured rather than inferred from the imagined objective.