MuZero and Value-Equivalent Models › MuZero and Value-Equivalent Models

Picture a video game. The screen is a torrent of pixels: explosions, particle effects, a scrolling scoreboard, a blinking cursor in the corner. Now picture the decision the game asks of you: move left, jump, or wait. Much of that pixel torrent may not matter for the decision. What matters is a task-relevant summary: where am I, where is the enemy, how much health do I have, is the platform solid? A game-playing agent must learn which details to retain for its decisions, and its training objective influences what its internal representation keeps.

A prominent visual-model lineage learns to reconstruct observations alongside its decision model. PlaNet and the Dreamer family, discussed in Part VIII: Decision-Centric Research Lineages, use a reconstruction objective to help train their latent states. That objective can spend capacity on observation details that do not change an action. This is one approach to model-based reinforcement learning, not its historical definition; classical control models often operate directly on compact physical states.

MuZero, introduced by Schrittwieser and colleagues in 2019 and refined in 2020, asked a sharper question. What if we skip the reconstruction entirely? What if the only requirement on the learned model is that it supports good decisions?

This decision-centric answer became highly influential. MuZero learns three connected functions:

• a representation function that turns an observation into a latent state,
• a dynamics function that advances a latent state under an action and predicts the reward,
• a prediction function that reads a policy and a value out of a latent state.

Here, a "latent state" means an internal representation the network learns, with no requirement that it reconstruct the raw observation. The reward is scalar feedback from the environment, and the policy is a distribution over actions. These three functions form MuZero's learned model; self-play data-collection workers, a replay buffer, and tree search are also essential parts of the agent.

There is no observation decoder or reconstruction loss in the original MuZero objective. Planning uses Monte Carlo tree search over the learned latent dynamics. Schrittwieser et al. reported strong results in Go, chess, shogi, and Atari without giving the model the games' transition rules. The paper's large-data Atari comparison used 20 billion frames for MuZero and 37.5 billion for R2D2; its 200-million-frame MuZero Reanalyze setting is a separate experiment. Neither result supports a blanket claim of a hundredfold or thousandfold sample-efficiency improvement.

This chapter builds a MuZero-style tutorial model on a small gridworld and tests latent prediction, tree search, and search-generated targets. The measurements also show limits to what these experiments establish: more search can worsen the planner's Q-greedy action agreement, while better value targets alone do not establish better learned-policy decisions. We will connect this to value equivalence, a formal criterion based on selected Bellman updates rather than pixel-level prediction.