Games and Learned Game Engines › Atari, Board Games, and Open Worlds

The word "game" hides wildly different problems. A world model that works on Atari will not automatically work on Go, and both will struggle in Minecraft for reasons that have nothing to do with one another. Part of the confusion is that "game" is defined by its user interface and cultural role, not by its mathematical structure. Two games that look similar on a screen can live in completely different regimes when characterized as dynamical systems. What separates them is a small set of structural properties: state dimension, observability, branching factor, horizon, determinism, and the amount of hidden mechanics. Understanding those properties is what turns "we trained a model that plays games" into a statement we can reason about. It is also what tells you, in advance, which techniques are likely to help and which are likely to stall.

The definition is deliberately loose on the word "plausible." It might mean a close match to the next frame, or only enough accuracy to support a specified decision. The relevant standard depends on the consumer: an agent needs decision-relevant quantities, while a player also needs a responsive, intelligible view.

The Arcade Learning Environment (ALE), introduced by Bellemare and colleagues, provides a common interface to hundreds of Atari 2600 environments; the familiar 57-game evaluation set is a benchmark selection, not the interface's limit. Standard RGB observations are typically 210 × 160 pixels, though game-specific observation spaces should be checked; the emulator exposes discrete joystick/button combinations, scores, and configurable evaluation choices. Those choices matter. Frame skipping, episode starts, and stochasticity can change the task being measured. In particular, the later ALE protocol study introduced sticky actions to make action execution stochastic and reduce exploitation of deterministic trajectories. A score is comparable across papers only after the game set and protocol have been specified.

A typical 210 × 160 ALE RGB frame contains 210 × 160 × 3 = 100,800 channel values, but image size is not the same as state complexity. The emulator can carry variables not recoverable from a single image, including timing and off-screen objects. Some Atari tasks therefore require a history of observations or another memory mechanism to predict what follows. A compact learned state should retain what matters for the intended task, rather than reconstructing every pixel; that is the sufficient-state question from Part III: Representing Agents and Worlds. Neither "all Atari is deterministic" nor "all Atari state is tiny" follows from the image dimensions.

The 2015 DQN result showed that a convolutional value function could learn control from processed game frames across multiple Atari titles, using the same algorithmic setup but separate training for each game. Its short frame history helps expose motion. Sparse-reward games such as Montezuma's Revenge remained difficult, however; that observation alone does not isolate representation, memory, or exploration as the cause. DQN is not itself a learned transition model, so a stack of frames should not be called a world model.

In chess, Go, and shogi, both players see the board and the legal moves are defined by known rules. The challenge is less hidden state than the number of possible continuations. A Go position can offer hundreds of legal moves, and search depth multiplies those choices. AlphaZero used known game rules with learned policy and value predictions to guide Monte Carlo tree search. MuZero kept search but learned the dynamics used inside it. Those are importantly different uses of a model: AlphaZero learns how to evaluate and prioritize positions, whereas MuZero learns a compact transition representation as well.

The value-equivalence lesson belongs to MuZero's learned search model, not to AlphaZero's known-rule transition function. MuZero trains its representation, dynamics, and prediction components for reward, policy, and value targets without requiring an image decoder. A latent model can be useful for search without faithfully reconstructing the visible board, but success on those targets does not mean it has learned every rule or is valid under every action sequence. We return to this distinction in Part VIII's planned chapter on MuZero and value-equivalent models.

Minecraft illustrates open-ended, compositional tasks: an agent may need to remember locations and combine objects, resources, and intermediate actions over a long horizon. DOOM is a different, level-based first-person example used in learned video generation; it should not be treated as an open-world benchmark simply because it has a rendered 3D view. Both can expose only part of the underlying state through the current image. A predictor trained on familiar scenes also needs to be tested on changed arrangements, as discussed in Compositionality and Systematic Generalization.

The difficulty of a Minecraft objective such as collecting diamonds includes sequential dependencies: gathering resources and crafting tools precede mining. A model or policy can fail at any dependency, so final reward by itself does not diagnose the failure. DreamerV3 reported collecting diamonds from scratch, but that result does not make every generated-world system an open-world agent. GameNGen modeled DOOM, DIAMOND trained an Atari agent with a diffusion world model and demonstrated a separate Counter-Strike: Global Offensive generator, and the Genie family pursued interactive generation. These are different tasks, data sources, and evaluation protocols.

It is worth tabulating the structural differences because they determine which techniques help and which do not. Two regimes that look superficially alike, both rendered as pixel grids, for instance, can differ in ways that force completely different architecture choices.

• Observation. An Atari image and a Minecraft view may omit state; a board position in chess or Go exposes the game pieces, with any clock modeled separately.
• Branching and horizon. Legal move counts, episode lengths, and useful planning horizons depend on the game and evaluation protocol. Go search and a multi-stage Minecraft task stress different kinds of depth.
• Stochasticity. Board-game rules are usually deterministic. ALE can be evaluated with stochastic sticky actions, while some 3D game environments, including procedurally generated Minecraft worlds, include random elements. A stochastic predictive distribution is useful when uncertainty affects decisions, not automatically mandatory for every open world.
• Rules. Chess rules are directly specified; a pixel-only learner must infer relevant game mechanics from experience. Rule consistency is a testable property only after the rule and observation conditions are stated.

A method demonstrated in one regime should therefore be evaluated again in the next, rather than assumed to fail or transfer automatically. The relevant tests change with the task: hidden-state inference for a pixel-only Atari agent, search quality for board play, and persistence and compositional behavior for a longer generated world.