The Dreamer Family › DreamerV3 and cross-domain scaling
World Models HandbookAll chapters The Dreamer Family › DreamerV3 and cross-domain scalingOne section · Sections in this chapter5 / 7 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Earlier Dreamer versions used configurations developed for narrower benchmark suites. DreamerV3 evaluates one reported configuration across Atari, DMLab, robot control, and a modified Minecraft task. This establishes configuration robustness across the tested suites, not held-out-domain generalization.
One configuration reduces per-environment tuning within the evaluated scope. It does not by itself rule out suite-level engineering or establish generalization to unseen domains.
The problem with cross-domain robustness is scale mismatch. Different domains have:
- Very different reward scales. Atari games often use small discrete game rewards, continuous-control tasks accumulate dense rewards over an episode, and the modified Minecraft task has sparse rewards. The resulting return and value targets can occupy different numerical ranges under one optimizer configuration.
- Very different observation statistics. Pixel observations are in , but proprioceptive observations, robot states, and image statistics vary widely. A model that is not scale-invariant will be biased toward some domains.
- Very different horizons. Some tasks are solved in tens of steps; others (like Minecraft) require thousands of steps with sparse reward.
The following DreamerV3 transforms mainly address numerical scale and optimization. Long-horizon exploration and model accuracy remain separate challenges.
Symlog transformation. DreamerV3 applies symlog to vector observations as both encoder inputs and decoder reconstruction targets, and uses symexp-spaced reward and value bins:
where:
- : the input value (e.g., a reward or value target) in the original scale
- : the sign of , preserving the direction of the reward or value
- : a compressing transform of the magnitude, so large magnitudes grow logarithmically rather than linearly
This map is monotone, sign-preserving, logarithmically compressive, and unbounded. For magnitudes well above one, multiplying by 100 changes symlog by approximately ; that approximation fails near zero. Its inverse is symexp.
Two-hot prediction. In the published DreamerV3 formulation, the reward and critic heads predict categorical distributions over original-scale bins . A scalar target assigns weights to its two neighboring values in , providing linear interpolation within that finite support. Decoding takes the probability-weighted average of the original-scale bin locations, ; it is not symexp applied after an average in symlog space. Targets outside the support require clipping, so the representation is not precise at arbitrary unknown scales.
Return normalization. DreamerV3 divides the actor advantage by an exponentially tracked 95th-to-5th percentile range of returns, floored at one. It does not rescale rewards or critic targets this way. The entropy and KL loss scales are fixed configuration values, not adaptive coefficients.
Free bits. DreamerV3 clips each of its split KL losses from below at one nat. This removes additional KL pressure below the threshold; it does not guarantee a floor on mutual information or prevent the stochastic channel from becoming uninformative.
Together with one-percent uniform mixing, architectural choices, replay, and optimizer settings, these techniques supported one reported configuration across DM Control, DMLab, Atari, and a modified MineRL Diamond task. The final Nature paper reports that every training run discovered diamonds without human data or task-specific curricula. This is a milestone under that protocol, not evidence that long-horizon exploration is solved.
The following standalone helpers demonstrate symlog, symexp, and two-hot encoding. They are not integrated into the scalar critic above.
def symlog(x):
return torch.sign(x) * torch.log1p(torch.abs(x))
def symexp(x):
return torch.sign(x) * torch.expm1(torch.abs(x))
def value_bins(num_bins=41, symlog_limit=20.0, device=None):
symlog_bins = torch.linspace(
-symlog_limit, symlog_limit, num_bins, device=device
)
return symexp(symlog_bins)
def two_hot(target, num_bins=41, symlog_limit=20.0):
# Interpolate the original-scale target between symexp-spaced bins.
bins = value_bins(num_bins, symlog_limit, target.device)
clipped = torch.clamp(target, bins[0], bins[-1])
upper = torch.searchsorted(bins, clipped).clamp(1, num_bins - 1)
lower = upper - 1
lower_bin, upper_bin = bins[lower], bins[upper]
upper_weight = (clipped - lower_bin) / (upper_bin - lower_bin)
lower_weight = 1.0 - upper_weight
out = torch.zeros(target.shape[0], num_bins, device=target.device)
out.scatter_(1, lower.unsqueeze(1), lower_weight.unsqueeze(1))
out.scatter_add_(1, upper.unsqueeze(1), upper_weight.unsqueeze(1))
return out
def two_hot_value(critic_logits, num_bins=41, symlog_limit=20.0):
# Average original-scale bins; split signs to reduce cancellation error.
probs = F.softmax(critic_logits, dim=-1)
bins = value_bins(num_bins, symlog_limit, critic_logits.device)
weighted = probs * bins
midpoint = num_bins // 2
negative = weighted[..., :midpoint].flip(-1).sum(dim=-1)
if num_bins % 2:
center = weighted[..., midpoint]
positive = weighted[..., midpoint + 1 :].sum(dim=-1)
else:
center = torch.zeros_like(negative)
positive = weighted[..., midpoint:].sum(dim=-1)
return negative + center + positiveNow verify the symlog round trip and observe its logarithmic compression. This check does not imply scale invariance.
sample_rewards = torch.tensor([-100.0, -1.0, 0.0, 1.0, 100.0, 10000.0])
compressed = symlog(sample_rewards)
round_trip = symexp(compressed)Max round-trip error after symlog/symexp: 2.93e-03 Values from 1 to 10000 span four orders of magnitude before compression.
For inputs restricted to , symlog outputs lie near . Symlog itself remains unbounded. Two-hot prediction is also limited by its chosen finite bin support.
A central engineering detail in the published DreamerV3 formulation is the combination of symlog vector inputs, reward and value predictions over symexp-spaced original-scale bins, and percentile-normalized advantages. Its actor uses REINFORCE with an entropy term and stopped world-model gradients. The reported configuration does not use per-environment reward scaling or discount tuning.
We will not re-train DreamerV3 here. These transforms are individually compact, but the benchmark result belongs to their tested combination with the categorical RSSM, uniform mixing, split KL objective, architecture, optimizer, and replay design. Adding three transforms to V1 does not establish the same robustness.
Several limitations of DreamerV3 deserve attention, because a chapter that only reports the wins is not honest:
- Sample efficiency does not imply lower compute. RSSM training and imagination add work, but wall-clock comparisons with model-free methods depend on architectures, replay ratios, hardware, and implementations.
- Minecraft remains protocol-specific. Diamond discovery in the modified MineRL setting is a notable result without human data or curricula, not a general solution to sparse long-horizon exploration.
- Imagination rewards can diverge from real rewards. Advantage normalization and free bits address numerical scale and KL pressure; they do not by themselves prevent out-of-distribution model exploitation.
- One configuration does not mean assumption-free. The method still depends on the observation/action interfaces, replay distribution, training budget, and an environment regular enough to learn predictive latent dynamics.
- Interpretability is not guaranteed. Learned latent coordinates need not align with human concepts, although probes or visualizations can sometimes reveal structure. A decoder tests reconstructive information, not semantic transparency.
These limitations animate a great deal of subsequent work. Conservative and offline variants (discussed later in this part) address distribution shift. Value-equivalent and control-centric models (the next two chapters) address the perception-versus-control gap by making the latent directly optimized for decision quality rather than reconstruction. Tokenized and video-generator world models (in Part IX) revisit the representation question with larger-scale generative machinery. And evaluation frameworks (in Part XI) provide the tools for systematically measuring whether an imagined policy performs well in the environment, rather than only appearing self-consistently optimistic.
Now let's see whether the Dreamer-style policy we trained earlier performs better than a random policy in the real environment. This summary uses the imagined return curve already logged during training, but samples a fresh set of 30 episodes per policy for the real-return comparison.
## Precompute the trained-policy and random-policy returns before the plot cell, so figures only render data.
rng_eval = np.random.default_rng(2024)
random_returns = []
for _ in range(30):
pos, vel = world.reset(rng_eval)
r = 0.0
for _ in range(50):
action = rng_eval.integers(0, 4)
pos, vel, reward, done = world.step(pos, vel, action, rng_eval)
r += reward
random_returns.append(r)
trained_returns = []
with torch.no_grad():
for _ in range(30):
pos, vel = world.reset(rng_eval)
r = 0.0
deter = model.initial_state(1, torch.device("cpu"))
prev_action = torch.zeros(1, model.action_dim)
prev_latent = torch.zeros(1, model.latent_dim)
for _ in range(50):
img = world.render(pos, 16)
obs_t = torch.tensor(img, dtype=torch.float32).unsqueeze(0)
deter, _, (q_mean, _) = model.forward_step(
prev_action, prev_latent, deter, obs_t.view(1, -1)
)
latent = q_mean
dist = actor(deter, latent)
action = int(dist.probs.argmax(dim=-1).item())
pos, vel, reward, done = world.step(pos, vel, action, rng_eval)
r += reward
prev_action = one_hot_action(
torch.tensor([action]), model.action_dim, torch.device("cpu")
)
prev_latent = latent
trained_returns.append(r)
imagined_curve = [h[3] for h in agent_history]
Together, the plots show a changing imagined lambda-return diagnostic and measured real-environment return. Neither improves in this toy run: the imagined diagnostic decreases, and the trained greedy policy performs worse than the random baseline. These measurements use different horizons, seed states, and action-selection rules, so they cannot establish a transfer failure or identify why the policy underperforms.
PlaNet used an RSSM for online planning from pixels, building on earlier recurrent latent-state models. DreamerV1 retained PlaNet's RSSM-based world model and trained an actor and critic on imagined latent trajectories. DreamerV2 combined categorical states, KL balancing, and generation-specific actor gradients for the full Atari suite and continuous control. DreamerV3 evaluated a broader package (symlog vector encoder inputs and decoder targets, symexp two-hot reward and value heads, advantage normalization, free bits, uniform mixing, architecture, replay, and optimizer choices) with one reported configuration across several benchmark suites.
The next chapter on MuZero takes a different stance. MuZero learns latent dynamics that predict reward, value, and policy targets. When selecting actions in the real environment, its Monte Carlo tree search simulates transitions with that learned model rather than querying the environment for hypothetical outcomes.