MuZero and Value-Equivalent Models › Summary

MuZero trains a model on planning-relevant reward, value, and policy targets and evaluates the resulting decisions. Its architecture has three functions: a representation $h_\theta$ that encodes available history into an initial latent state (our tutorial uses $s^0 = h_\theta(o_t)$), a dynamics $g_\theta$ that advances latent states under actions and predicts rewards via $(s^{k+1}, r^{k+1}) = g_\theta(s^k, a^k)$, and a prediction $f_\theta$ that reads a policy $p^k$ and a value $v^k$ out of a latent state via $(p^k, v^k) = f_\theta(s^k)$. There is no decoder and no reconstruction loss in the original objective.

• The original MuZero latent is not supervised to reconstruct observations. Reward, value, and policy targets constrain its decision-relevant predictions; the tutorial adds an auxiliary consistency loss and uses only one-step supervision.
• Latent consistency is optional. Original MuZero does not include it; EfficientZero and our tutorial use variants. The toy probe shows reduced but nonzero distractor information, so it did not eliminate all four noise bits.
• Planning happens in latent space. Search uses predicted edge rewards and child values alongside priors. In this learned model, increasing the simulation budget worsened Q-greedy action agreement while improving root-value accuracy; that diagnostic does not directly measure visit-policy action selection.
• Value equivalence is relative to selected tests. Grimm et al. define it through Bellman-operator agreement for selected policies and value functions, not blanket equality of all policy values. Reconstruction and decision-target objectives preserve different information.
• Search targets need careful use. Published MuZero uses search policy targets plus outcome or bootstrapped-return value targets; Reanalyze refreshes selected policy targets. In this toy experiment, jointly changing search-value and search-policy targets improves equally weighted cell-mean head RMSE from 0.104 to 0.072. A separate replay-matched check gives raw-search and tuned-head RMSE of 0.040 and 0.039, respectively; those replay rows overrepresent the goal.
• The failure modes are real. Uninterpretable latents, model exploitation through overestimation, expensive search, and an assumption of deterministic dynamics all require care. The lineage that followed MuZero, from EfficientZero to Sampled and Gumbel MuZero, is largely a catalogue of responses to these specific weaknesses.