Games and Learned Game Engines

Press right in a grid world, and the agent should move one cell unless a boundary stops it. A conventional engine implements that rule directly. A learned engine predicts the next observation from examples of actions and outcomes; it may produce the right picture while getting the rule wrong. That difference matters both to a player who expects a coherent game and to an agent planning through predicted transitions. This chapter uses a small learned grid-world engine to measure the gap between one-step prediction, longer rollouts, and decisions made with the model.

Games are useful testbeds because an evaluator can often reset them, choose actions, and observe rewards. But the game families below pose different problems. A single Atari frame can omit emulator state; a board position in Go exposes the pieces and legal moves; a generated first-person scene must remain responsive and recognizable across actions. No one architecture or metric solves all three.

We will compare Atari, board games, and more open-ended interactive worlds; distinguish models used for agent decisions from worlds rendered for people; and test prediction, rollout error, and model-predictive control in one executable example. A compact evaluation checklist below separates controllability, visual quality, rule fidelity, and decision usefulness; it is not a new benchmark. The chapter builds on Part V: World-Model Architectures and Part VIII: Decision-Centric Research Lineages, before the application domains that follow.

The word "game" hides wildly different problems. A world model that works on Atari will not automatically work on Go, and both will struggle in Minecraft for reasons that have nothing to do with one another. Part of the confusion is that "game" is defined by its user interface and cultural role, not by its mathematical structure. Two games that look similar on a screen can live in completely different regimes when characterized as dynamical systems. What separates them is a small set of structural properties: state dimension, observability, branching factor, horizon, determinism, and the amount of hidden mechanics. Understanding those properties is what turns "we trained a model that plays games" into a statement we can reason about. It is also what tells you, in advance, which techniques are likely to help and which are likely to stall.

The definition is deliberately loose on the word "plausible." It might mean a close match to the next frame, or only enough accuracy to support a specified decision. The relevant standard depends on the consumer: an agent needs decision-relevant quantities, while a player also needs a responsive, intelligible view.

The Arcade Learning Environment (ALE), introduced by Bellemare and colleagues, provides a common interface to hundreds of Atari 2600 environments; the familiar 57-game evaluation set is a benchmark selection, not the interface's limit. Standard RGB observations are typically 210 × 160 pixels, though game-specific observation spaces should be checked; the emulator exposes discrete joystick/button combinations, scores, and configurable evaluation choices. Those choices matter. Frame skipping, episode starts, and stochasticity can change the task being measured. In particular, the later ALE protocol study introduced sticky actions to make action execution stochastic and reduce exploitation of deterministic trajectories. A score is comparable across papers only after the game set and protocol have been specified.

A typical 210 × 160 ALE RGB frame contains 210 × 160 × 3 = 100,800 channel values, but image size is not the same as state complexity. The emulator can carry variables not recoverable from a single image, including timing and off-screen objects. Some Atari tasks therefore require a history of observations or another memory mechanism to predict what follows. A compact learned state should retain what matters for the intended task, rather than reconstructing every pixel; that is the sufficient-state question from Part III: Representing Agents and Worlds. Neither "all Atari is deterministic" nor "all Atari state is tiny" follows from the image dimensions.

The 2015 DQN result showed that a convolutional value function could learn control from processed game frames across multiple Atari titles, using the same algorithmic setup but separate training for each game. Its short frame history helps expose motion. Sparse-reward games such as Montezuma's Revenge remained difficult, however; that observation alone does not isolate representation, memory, or exploration as the cause. DQN is not itself a learned transition model, so a stack of frames should not be called a world model.

In chess, Go, and shogi, both players see the board and the legal moves are defined by known rules. The challenge is less hidden state than the number of possible continuations. A Go position can offer hundreds of legal moves, and search depth multiplies those choices. AlphaZero used known game rules with learned policy and value predictions to guide Monte Carlo tree search. MuZero kept search but learned the dynamics used inside it. Those are importantly different uses of a model: AlphaZero learns how to evaluate and prioritize positions, whereas MuZero learns a compact transition representation as well.

The value-equivalence lesson belongs to MuZero's learned search model, not to AlphaZero's known-rule transition function. MuZero trains its representation, dynamics, and prediction components for reward, policy, and value targets without requiring an image decoder. A latent model can be useful for search without faithfully reconstructing the visible board, but success on those targets does not mean it has learned every rule or is valid under every action sequence. We return to this distinction in Part VIII's planned chapter on MuZero and value-equivalent models.

Minecraft illustrates open-ended, compositional tasks: an agent may need to remember locations and combine objects, resources, and intermediate actions over a long horizon. DOOM is a different, level-based first-person example used in learned video generation; it should not be treated as an open-world benchmark simply because it has a rendered 3D view. Both can expose only part of the underlying state through the current image. A predictor trained on familiar scenes also needs to be tested on changed arrangements, as discussed in Compositionality and Systematic Generalization.

The difficulty of a Minecraft objective such as collecting diamonds includes sequential dependencies: gathering resources and crafting tools precede mining. A model or policy can fail at any dependency, so final reward by itself does not diagnose the failure. DreamerV3 reported collecting diamonds from scratch, but that result does not make every generated-world system an open-world agent. GameNGen modeled DOOM, DIAMOND trained an Atari agent with a diffusion world model and demonstrated a separate Counter-Strike: Global Offensive generator, and the Genie family pursued interactive generation. These are different tasks, data sources, and evaluation protocols.

It is worth tabulating the structural differences because they determine which techniques help and which do not. Two regimes that look superficially alike, both rendered as pixel grids, for instance, can differ in ways that force completely different architecture choices.

• Observation. An Atari image and a Minecraft view may omit state; a board position in chess or Go exposes the game pieces, with any clock modeled separately.
• Branching and horizon. Legal move counts, episode lengths, and useful planning horizons depend on the game and evaluation protocol. Go search and a multi-stage Minecraft task stress different kinds of depth.
• Stochasticity. Board-game rules are usually deterministic. ALE can be evaluated with stochastic sticky actions, while some 3D game environments, including procedurally generated Minecraft worlds, include random elements. A stochastic predictive distribution is useful when uncertainty affects decisions, not automatically mandatory for every open world.
• Rules. Chess rules are directly specified; a pixel-only learner must infer relevant game mechanics from experience. Rule consistency is a testable property only after the rule and observation conditions are stated.

A method demonstrated in one regime should therefore be evaluated again in the next, rather than assumed to fail or transfer automatically. The relevant tests change with the task: hidden-state inference for a pixel-only Atari agent, search quality for board play, and persistence and compositional behavior for a longer generated world.

A learned simulator replaces some or all hand-coded transitions with a model trained on interaction data. A frame predictor outputs a next image from recent images and actions; an interactive generator must also keep responding over many steps. Both are world models, but the right architecture and tests depend on whether the output is used for planning, training, or play.

The purpose matters, because it changes which errors are consequential. Two uses are useful to distinguish.

• Simulation for agents. The model generates transitions used for planning or imagined policy updates. Reward, value, termination, and action consequences must be accurate enough for the chosen decision process; photorealistic reconstruction is optional. Dreamer and MuZero are examples with learned latent dynamics.
• Playable generation for humans. The system renders an action-responsive view under a latency budget. Visual continuity, controllability, and rule behavior matter together. GameNGen and the Genie systems are examples, but their tasks and hardware differ.

The representation follows that objective. A compact latent can reduce the cost of simulating candidate futures, but its efficiency depends on model size, rollout length, hardware, and any decoder or search. A renderable system may still predict compressed visual tokens rather than raw pixels. Ha and Schmidhuber's World Models is already a hybrid: it learns visual compression and latent dynamics, then reconstructs frames for inspection. The practical comparison is task- and implementation-specific, not an overnight-versus-month rule.

Building on Recurrent State-Space Models, the latent lineage encodes observations and predicts future latent states under actions. PlaNet combines a deterministic recurrent path with stochastic latent variables. Dreamer uses imagined latent trajectories to improve its actor and critic, while collecting additional environment interactions to train and update the world model. DreamerV2 and V3 extended that line. A stochastic latent can represent multiple possible futures, though deterministic models remain useful when their assumptions fit the task. With $r_t$ denoting reward after action $a_t$ and $c_t$ denoting whether the episode continues after that action, one generic transition description is

where:

• $z_t$: a latent representation at time $t$ that may summarize observation history
• $a_t$: the action taken at time $t$
• $p(z_{t+1} \mid z_t, a_t)$: the learned transition distribution over the next latent given the current latent and action
• $r_t$: the reward after action $a_t$, whose value the model predicts during imagination
• $c_t$: the continuation indicator after action $a_t$ (one if the episode continues, zero if it terminates)
• $p(r_t,c_t \mid z_t,a_t,z_{t+1})$: a generic reward-and-continuation prediction; individual implementations may condition these heads on only a subset of the listed variables

The sampled transition can represent more than one possible next latent for the same inputs. Whether the latent should be interpreted as a belief over hidden states depends on the inference model and observation history; stochasticity alone does not establish that interpretation. The indexing above keeps the after-action reward aligned with the worked example below.

MuZero trains representation, dynamics, and prediction networks for search targets without an observation-reconstruction objective. Its learned latent need not depict the board or Atari frame. That can avoid spending model capacity on visual details irrelevant to its search targets; it does not prove a general compute advantage over every reconstructive model. Search cost still depends on model architecture and the number of nodes expanded.

The relevant intuition is decision sufficiency: a model need not reconstruct everything to support a particular search objective. If the task only requires choosing an action in two visually different states, preserving the action ranking can suffice. Equal action-conditional rewards and values are a stronger sufficient condition, not a necessary one. A new task or action set may require distinctions the representation discarded.

Ha and Schmidhuber's World Models (2018) combined a VAE for image compression, an MDN-RNN for latent dynamics, and a small controller. The VizDoom controller was trained in a generated dream environment; the CarRacing experiment used a different training arrangement. The result is an influential early hybrid of learned compression and dynamics.

The visual-generation projects differ in both training data and output. DIAMOND (2024) used a diffusion world model to train an Atari agent and separately demonstrated an interactive model trained on static Counter-Strike: Global Offensive gameplay. GameNGen (2024) first recorded DOOM play from an RL agent, then trained an action-conditioned diffusion frame predictor; its reported interactive rate was about 20 frames per second on a single TPU. Genie (2024) learned latent actions from unlabeled video for controllable generation. That training description should not be copied verbatim onto every successor. Genie 2 generated action-controlled 3D scenes for up to a minute in reported examples, most of which lasted 10–20 seconds. More recently, Genie 3 reported real-time 720p interaction at 24 frames per second with consistency over a few minutes. These are reported demonstrations under different settings, not matched benchmark scores.

The differences between these methods come down to a small set of choices.

• Observation representation. Pixels, latent vectors, discrete tokens. Each has different compression efficiency and different reconstruction quality.
• Dynamics model. Recurrent latent prediction (Dreamer), token prediction (for example, IRIS), or diffusion-based visual prediction (DIAMOND and GameNGen).
• Conditioning data. Recorded actions are available for GameNGen's gameplay data; Genie learns a latent action representation from video without action labels.
• Temporal context. A longer input history can help with hidden state or visual continuity, but increases memory and compute in architecture-dependent ways.

These choices interact, but none fixes long-horizon behavior automatically. The right design depends on whether the output is used for decisions, rendered to a player, or evaluated as a research model.

Playable generation places stricter human-facing requirements on three properties that can also matter to an agent-facing simulator.

• Controllability. Actions must causally change the generated world, with feedback fast enough for the intended interaction. A game that ignores a button press is not playable, regardless of visual quality.
• Interactive throughput. Latency must be low enough for the intended control loop. GameNGen reports about 20 frames per second on a single TPU; that is a measured configuration, not a universal threshold.
• Persistence. Revisiting a location should reveal a coherent continuation of the earlier scene when the task requires it. A fixed context window may make that difficult, but its limit must be measured for the particular model.

One-step frame quality alone does not establish playability. A practical evaluation should record the model, hardware, frame rate and input-to-output latency; test whether changing an action changes the next outcome; compare multi-step trajectories and revisited locations against a reference when one exists; count specified rule violations; and, if agent training is the goal, evaluate the resulting policy in the reference environment. Short attractive clips and downstream return answer different questions. None of these measurements by itself certifies a complete game engine.

We will return to memory and consistency in the next section. For now, the takeaway is that learned simulators are not one thing. They are a family of compromises between fidelity, speed, controllability, and abstraction, and the right point on that surface depends on whether the consumer is an agent computing gradients or a human holding a controller. Choosing the wrong point can make an otherwise excellent model useless for your purpose, which is why the "what is it for" question has to be answered first.

We will now build a $5 \times 5$ grid-world engine with a movable agent and fixed goal. Its two-channel observation marks the agent and goal, and its after-action reward is negative Manhattan distance to the goal. A small network predicts the next observation and reward. We will test how its predictions change under recursion, then use it to score candidate action sequences while executing and evaluating the resulting choices in the reference environment.

The observation contains 50 binary entries: two 5 × 5 channels, one for the agent and one for the goal. That makes mistakes easy to inspect. The experiment is a teaching example, not a timing or scaling result for pixel-based engines.

We import the scientific stack and set NumPy and PyTorch seeds so this notebook run can be reproduced in a compatible environment. A seed does not remove sampling uncertainty, implementation differences, or training-data effects; later curves describe one seeded run.

Each transition is $(o_t,a_t,o_{t+1},r_t)$, where $r_t$ is the reward after action $a_t$ and $o_t \in \{0,1\}^{2\times5\times5}$. The model maps a flattened observation and one-hot action to a predicted next observation and scalar reward. Unlike Dreamer, it has no recurrent latent state. It is deliberately small enough to inspect, and it can still expose the same gap between one-step prediction and recursive use.

The architecture is minimal. A two-hidden-layer MLP takes the flattened observation and one-hot action; separate linear heads predict the next observation and reward. The heads allow distinct losses or weights, although this example simply sums two mean-squared errors. Other world models may use different state representations, objectives, and rollout tests; this MLP is not a template for all of them.

The training objective falls, but it cannot by itself tell us how well the model predicts held-out transitions. The environment is deterministic, while the training targets include small random perturbations. The next tests separate fitting this objective from one-step prediction and recursive rollout fidelity.

We start with 5,000 validation transitions set aside from the end of the collected array. This is a transition-level split: the boundary need not be episode-disjoint, and the small grid permits repeated states. We compute the mean Euclidean (L2) distance between flattened predicted and true next observations:

over that held-out set of $N$ transitions. Here:

• $N$: the number of held-out transitions evaluated
• $\hat{o}_{t+1}^{(i)}$: the model's predicted next observation for the $i$-th held-out transition, flattened to a vector
• $o_{t+1}^{(i)}$: the true next observation for the $i$-th held-out transition, flattened to a vector
• $\| \cdot \|_2$: the Euclidean (L2) norm, so each example contributes the overall pixel-space distance between prediction and truth

The errors cluster around an L2 distance of one on the flattened two-channel grid, not around zero. That is a meaningful one-step mismatch in this small environment: the clustered error should not be mistaken for near-exact predictions. One-step validation remains useful, but it does not reveal how errors change when predictions become inputs to later predictions. We therefore measure recursive rollout error separately.

One-step accuracy is not enough when a model is fed its own predictions. A small error may change the next input, which can create further error; whether the deviation grows, shrinks, or changes decisions depends on the learned transition and the states visited. The bound below shows one possible relationship between one-step and recursive errors under an explicit smoothness assumption. It is not a universal forecast that every video model fails after a fixed number of frames.

Suppose the true dynamics are $s_{t+1}=f(s_t,a_t)$ and a model rollout is $\hat{s}_{t+1}=\hat f(\hat s_t,a_t)$ under the same action sequence. Let $e_t=\|\hat s_t-s_t\|_2$ and let $\epsilon_t=\|\hat f(s_t,a_t)-f(s_t,a_t)\|_2$ be the one-step model error evaluated at the true state. If $\hat f$ is $L$-Lipschitz in its state input along both compared trajectories, the triangle inequality gives

where:

• $e_t$: the error $\|\hat{s}_t - s_t\|$ between the predicted and true state at rollout step $t$
• $L$: the Lipschitz constant of $\hat{f}$ with respect to its state input, i.e., the worst-case factor by which a small state perturbation is amplified in one model step
• $\epsilon_t$: the one-step model error at step $t$, i.e., the error introduced when the model predicts the next state from the true current state

Unrolling the recursion gives the error at horizon $T$:

Here $e_0$ is the initial error. If every $\epsilon_t\le\epsilon$, then for $L<1$ the bound approaches $\epsilon/(1-L)$ as $T$ grows while the initial-error term decays; for $L=1$ it is $e_0+T\epsilon$; and for $L>1$ it may grow exponentially. This is an upper bound, not an equality or a measured growth law. Its premises require the same actions and a Lipschitz bound covering the compared states, including the model-generated ones; a bound measured only on training states would not suffice for an out-of-distribution rollout.

The three regimes can be compared directly by iterating the recursion with a constant per-step error.

The analytical curves illustrate possible amplification. They do not estimate this fitted grid-world model's Lipschitz constant. Shortening a rollout limits the number of recursive predictions, and re-anchoring replaces a predicted state with an observation; whether either improves decisions still requires evaluation.

Two useful descriptions of long-horizon failure are gradual drift and abrupt rule failure. They are not exhaustive or tied to a universal time scale.

• Drift. A predicted trajectory can gradually diverge in position, appearance, or timing while each frame still looks plausible. Measure it against a reference trajectory under a specified action sequence; do not infer a fixed failure frame count from a single demonstration.
• Rule failure. A model may generate a transition that violates a specified mechanic, such as a character jumping again in mid-air when the game permits only one jump. The error may appear abruptly or after a long rollout; it need not be caused only by an out-of-distribution input.

Different measurements reveal different problems. A frame-distance curve may reveal accumulating prediction error, while a targeted rule test can expose a failure that looks visually minor. Noise augmentation, broader training coverage, longer context, and explicit constraints are possible interventions, but their effects depend on the model and rule being tested.

Diffusion does not make temporal errors disappear: an autoregressive diffusion world model still conditions future frames on generated history. DIAMOND's project report shows a concrete rule violation—repeated jumps not allowed by the reference game. GameNGen used conditioning augmentations for stable autoregressive generation, and its authors evaluated clips after five minutes of generation. Neither example supports a blanket claim that visual generators fail after 30–60 frames or that one model family is intrinsically immune to collapse.

Games have mechanics that can be stated and tested. A learned model that lets an agent pass through an impassable wall may support a policy that fails in the reference game. But scores can legitimately decrease in some games, and a visible inconsistency does not guarantee that a trained agent will exploit it. To assess rule consistency, specify the rule, action distribution, and observation conditions, then count violations over evaluated transitions.

This idea is worth stating as a criterion.

Downstream agent return in the reference game is a separate, indirect test of decision usefulness: poor transfer could stem from rule errors, reward errors, policy optimization, or another mismatch. It cannot by itself identify which mechanic failed. Direct tests can replay a chosen action sequence and score specified mechanics. Part XI's planned chapter on state, physics, causality, and memory evaluation develops that distinction further.

Several techniques can be tested for longer rollouts; none is universal.

• Exposure to predicted states. Training on some model-generated histories can reduce the mismatch between training and rollout inputs, if those histories are representative. Do not assume every cited system uses the same procedure.
• Conditioning augmentation. Corrupting conditioning frames during training may improve robustness to prediction noise; GameNGen reports this intervention. It can also change one-step accuracy, so both should be measured.
• Temporal context or external memory. Access to earlier observations can help recover hidden or revisited details, at a model-dependent compute cost. A demonstrated one-minute video is not evidence of a one-minute input context window. See Temporal State and Memory.
• Latent prediction. A compact latent can omit pixel detail irrelevant to a decision objective; it does not make state or reward prediction error vanish.
• Re-anchoring. Replacing a predicted state with a new observation truncates the executed model rollout. It cannot retroactively remove errors in the candidate futures that selected the preceding action.

In the mathematical bound, a shorter horizon reduces the number of propagated terms. The other techniques may change the local errors or dynamics but do not guarantee a smaller $\epsilon_t$ or $L$. They need before-and-after evaluation under the same protocol.

We now collect fifty new random-policy episodes with a separate seed and compare model-generated observations with the reference environment under the same recorded actions, for at most twenty steps. Episodes terminate at different times, so the number contributing to each horizon can change. This is a fresh-episode test, unlike the transition-level validation split above.

The natural questions are: can an agent still use this model to make decisions, and how careful must it be about the drift? The next section addresses that.

A learned model can supply imagined trajectories for policy updates or candidate-action search, but the model itself first needs data. Dreamer continues to collect environment experience while learning from imagined trajectories; MuZero learns dynamics useful for search; Ha and Schmidhuber tested a dream-trained controller in VizDoom. GameNGen's DOOM generation is primarily a playable-model demonstration, not evidence that an agent trained in it transferred to the reference game. Computational and data advantages depend on the environment, hardware, model, and evaluation protocol.

A model used for decisions needs at least three properties under the intended task distribution.

1. Decision sufficiency. It should retain information needed to rank actions for the objective, even when visual reconstruction is not required.
2. Rollout stability over the planning horizon. The drift we just measured must be small enough that short plans remain accurate.
3. Reward realism. The predicted reward must be aligned with the true reward on states the planner tends to visit.

These conditions differ from visual playability, rather than forming a guaranteed sufficient test. A model can be useful for one planner while looking poor, and a rendered world can look persuasive while giving wrong rewards. Even when the three conditions appear to hold on a test set, distribution shift under optimization can expose further errors.

The literature separates two styles of use.

• Policy learning in imagination. The agent updates an actor and critic from modeled trajectories, as Dreamer does. The learned policy can exploit errors in the world model; actual training also needs environment data for that model.
• Planning in the model. A planner evaluates candidate futures and chooses an action, as in MuZero's MCTS or the simple MPC below. Replanning from a new observation limits the length of the executed open-loop plan but does not make the first decision immune to errors in simulated futures.

Both styles can exploit errors. An imagined policy is trained against model predictions; a planner can choose a bad first action because an inaccurate later reward made its candidate sequence look attractive. Receding-horizon search then observes the new real state before choosing again, while an actor does not necessarily search anew at each decision. Search also has a per-decision cost that depends on its budget. The example below demonstrates random-shooting MPC, not a general superiority claim for planning.

Optimization can favor model-predicted outcomes that do not occur in the reference environment. For the grid world, separate the true reward rule from the network's reward-head prediction. The environment returns

Here $s_{t+1}$ is the true post-action agent coordinate, $g$ is the true goal coordinate, and $\lVert\cdot\rVert_1$ is Manhattan distance. The learned model instead predicts

where $R_\theta$ is the separate scalar reward head after the MLP trunk. It is not computed by decoding the predicted next observation or measuring its distance to a decoded goal. In particular, a one-cell observation-prediction error does not imply a one-unit error in $\hat r_t$. The planner below sums these direct reward-head outputs. Model exploitation remains possible if a candidate action sequence receives optimistic reward predictions, but this code does not demonstrate that it occurs.

The distinction between the two heads matters diagnostically: observation drift and reward error need separate measurements. A visually wrong rollout might still rank actions well, while a visually plausible one might misrank them. Repeated optimization can search for optimistic predictions, but whether that happens here must be measured against true returns.

Common responses to this risk change the model, planner, or objective:

• Ensembles and uncertainty penalties. Model disagreement can signal poorly supported actions, although disagreement is not a perfect error estimate. PILCO, PETS, Ensembles, and MBPO compares related approaches.
• Short imagination horizons. Dreamer imagines finite rollouts and uses a value model to estimate returns beyond them. Shorter rollouts limit how far the learned dynamics are applied directly, although accumulated model error is not the paper's stated reason for this design.
• Real-data correction. New reference-environment transitions can reveal and repair model errors in visited regions, at an interaction cost.
• Conservatism. MOPO subtracts a model-uncertainty penalty from imagined rewards; MOReL routes unsupported state-actions to a pessimistic unknown state. Part VIII's planned chapter on offline and conservative model-based RL covers this family.

Each mitigation changes what the agent is allowed to optimize or what evidence it receives. None guarantees transfer for an arbitrary learned model. Evaluate the chosen policy in the reference environment, not solely against its training model.

At each environment step, random-shooting MPC samples action sequences of length $H$, rolls them out inside the learned model, and selects the sequence with the highest sum of predicted reward-head outputs. It executes only that sequence's first action, observes the real next state, and plans again. Re-anchoring prevents errors in an unexecuted trajectory from becoming the next observed state, but errors at any simulated depth can still affect which first action wins the search.

All four tested horizons reach the goal on every one of the thirty seeded evaluation episodes for that horizon. Each evaluation call starts its environment and planner generators from the same seed; different horizon lengths group and consume the planner draws differently, so this is not a paired comparison of identical candidate sequences. These saturated results show that the learned model supports this planner on these small test cases despite observation rollout error. They do not show a benefit from increasing the horizon, a drift-induced decline, an optimal horizon, or guaranteed improvement from more samples. On a harder task, those effects would need a broader paired evaluation.

Pixel-level rollout error and decision usefulness answer different questions. Here the planner reaches the goal on the tested episodes despite substantial observation error; thirty trials per horizon are too narrow to establish reliability elsewhere. Evaluate both model predictions and closed-loop outcomes on held-out starts and, where possible, new tasks, reporting uncertainty when comparing methods.

The cited systems establish different accomplishments under different tests. DreamerV3 reports results across more than 150 tasks, not blanket superhuman performance on all 57 Atari games. AlphaZero used known rules for board-game search, and MuZero learned search dynamics. GameNGen demonstrated action-responsive DOOM generation on specified hardware; Genie 2 and Genie 3 reported increasingly long interactive visual worlds. Those claims should be read with their game sets, protocols, hardware, and time horizons attached. They do not imply that one learned engine handles every game or that model-generated experience removes the need for reference-environment tests.

The limitations are equally real, and it is worth being honest about them. Four stand out.

First, evaluation conditions vary even within games. A deterministic board game, Atari with sticky actions, and Minecraft do not share one observability or reward structure. Success in one setting does not establish transfer to physical manipulation or driving, where sensing, action, contact, safety, and data collection impose different constraints. The remaining Part X: Application Domains chapters treat those settings separately.

Second, consistency depends on what is being preserved and for how long. GameNGen reported multi-minute DOOM trajectories, Genie 2 reported worlds up to a minute with most demonstrations shorter, and Genie 3 reported real-time consistency for a few minutes. These are important but not equivalent to arbitrary-length persistence of every object, rule, and reward. Conditioning augmentation, longer context, memory, and re-anchoring incur different costs; no universal doubling or quadrupling factor follows from their names.

Third, optimizing against an imperfect model can favor prediction errors. Uncertainty penalties and short rollouts can reduce particular risks, but a policy still needs evaluation outside the model. The toy MPC result above is a useful reminder: high closed-loop success on a small seeded set does not measure exploitation pressure across other starts, goals, or planners.

Fourth, no single metric covers all intended uses. PSNR compares images, distributional video metrics compare sampled videos, targeted tests count rule violations, and closed-loop return tests whether decisions transfer under a specified protocol. The evaluation checklist earlier keeps these questions separate. Part XI: Evaluation and Understanding develops the broader evaluation framework.

The useful conclusion is narrower than a single progress ladder: learned dynamics can support search, imagined policy updates, or interactive rendering, but each success is conditional on its task and measurement. Before transferring a method to another domain, identify which state, action, reward, latency, and persistence assumptions change.

Atari, board games, and open-ended interactive environments stress different properties: hidden state in pixel observations, branching search under known rules, and composition or persistence over longer tasks.

• Atari challenges models to retain action-relevant state from high-dimensional, partially observable observations.
• Board games distinguish AlphaZero's search with known rules from MuZero's learned, decision-oriented dynamics.
• Long-horizon open-world tasks often stress compositionality and memory, but the requirements depend on the task.

Latent dynamics are central to PlaNet, Dreamer, and MuZero; visual generation is central to DIAMOND, GameNGen, and Genie. World Models combined visual compression with latent prediction. The design trade-offs include decision targets, renderability, action conditioning, context, and measured compute rather than a universal latent-versus-pixel speed ranking.

Longer recursive use requires measuring both prediction error and specified rule behavior. Under the displayed same-action and Lipschitz assumptions, the error bound is linear for $L=1$ and may grow exponentially for $L>1$; it does not fit a particular rollout. Conditioning augmentation, temporal context, memory, and re-anchoring are candidates for testing, not guaranteed fixes.

An imperfect model can still support useful decisions. Dreamer-style imagination updates and model-based search are distinct uses, and both can be misled by reward or transition errors. The grid-world planner succeeds on its seeded test set even though its observation predictions drift; that does not establish transfer beyond this example.

The next chapter turns to robotic manipulation, where contact, sensing, and data collection change what a useful model must predict. We should carry the evaluation habit forward, not assume that a game result transfers unchanged.

The key parameters for the grid-world learned engine are:

• size: Grid dimension. Controls the state space and the number of possible agent and goal positions.
• max_steps: Reference-environment episode budget. It caps collected episodes and recorded-action rollout evaluation; MPC candidate length is set separately by horizon.
• hidden: Width of the MLP trunk. Increasing it raises parameter count and compute per model step; effects on held-out accuracy must be measured.
• lr: Adam learning rate for the world model. An excessively high value can destabilize training; a very low one can slow optimization.
• n_val: Number of held-out transitions. Sets the size of the evaluation split.
• horizon: Random-shooting MPC planning horizon. Longer horizons evaluate more predicted steps and can expose planning to more model error; the amount and direction must be measured on the planner's candidate distribution.
• n_samples: Number of random action sequences evaluated per MPC step. More samples increase search work and may find a higher predicted score; true performance need not improve if reward predictions are wrong.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about games and learned game engines.