The Dreamer Family › The Dreamer Family

The PlaNet agent from the preceding chapter learned a compact latent dynamics model from pixels, then planned inside it with online trajectory optimization. At every environment step, PlaNet's cross-entropy method evaluates candidate action sequences, refits its sampling distribution to elite sequences, and executes the current-step mean action from the optimized distribution. More candidates and a longer horizon increase this decision-time cost. The search itself is transient, although the world model continues to learn from experience.

The Dreamer family, developed by Danijar Hafner and colleagues across a sequence of papers from 2019 to 2023, makes a decisive move that separates it from the PlaNet-style planners. Instead of using the learned model only as a forward simulator for planning, Dreamer uses it as a training ground for policies and value functions. The actor and critic are the learned parameters that carry the reusable skill. The model is still a latent dynamics model learned from sensory inputs, including pixels and, in DreamerV3, vector observations. But the agent does not search over action sequences at every step. It learns an actor (a policy) and a critic (a value function) by imagining trajectories inside the learned model, then acts by simply querying the actor. This is the essence of "learning behaviors by latent imagination". It is what makes Dreamer both fast at action time and capable of improving the behaviors it uses.

To see the computational shift, compare the controllers. PlaNet searches, evaluates, and compares action sequences before each action; Dreamer spends that control-learning compute during training and retains the result in actor parameters. PlaNet still trains its world model, so not all of its expense occurs at decision time. The distinction is between repeating a transient controller search and amortizing a rule for action selection.

This reframing matters more than it might first appear. PlaNet improves its learned model, but its controller remains an online optimizer. Dreamer also trains an actor and critic, so behavioral knowledge persists in parameters instead of being recomputed by a fresh action-sequence search. Both approaches can exploit errors in the model; an amortized policy does not automatically become conservative around uncertainty.

The family spans three major versions, and each one solves a problem the previous version exposed. DreamerV1 established the core recipe, DreamerV2 scaled it with categorical states and KL balancing, and DreamerV3 added cross-domain robustness techniques.

• DreamerV1 (Dream to Control; 2019 preprint, 2020 publication) established the core recipe: a recurrent state-space model (RSSM) for latent dynamics, imagined rollouts, and actor-critic learning through imagined dynamics. It reparameterized continuous actions and used straight-through gradients for discrete actions.
• DreamerV2 (Mastering Atari with Discrete World Models; 2021 publication) replaced Gaussian stochastic states with 32 groups of 32 categorical variables, introduced KL balancing, and used REINFORCE actor gradients on Atari while retaining dynamics gradients for continuous control.
• DreamerV3 appeared as a 2023 preprint and was published in Nature in 2025. It uses categorical states, split KL losses with free bits, one-percent uniform mixing, symlog vector inputs, symexp-spaced two-hot reward and value predictions, and percentile-normalized actor advantages. Image observations retain their convolutional path. Its fixed configuration was evaluated across a broad benchmark suite with benchmark-specific interfaces and budgets.

By the end of this chapter you should be able to explain why learning a policy in imagination differs from planning, how RSSM variants represent latent state, how their actor gradients differ, and what supports DreamerV3's cross-domain configuration robustness. We also build a compact Dreamer-style agent and compare its imagined training diagnostic with real-environment performance.

A note on how to read this chapter: the theory and the code are meant to be read together. We run a compact Dreamer-inspired agent end to end on a small "dot world," then define selected V2 and V3 components without training full variants. Try to keep the mapping between the math and the code in mind as you go, because the machinery of Dreamer is not exotic; it is a handful of well-understood components (a recurrent model, a variational latent, an actor, a critic) assembled in a careful way.