Tokenized Game and Environment Models

Picture a small arcade game. There is an agent in a room, a locked door on the far side, a key glinting in a corner, and a hazard that ends the run. You record hours of play and train a model to predict the next screen given the current screen and the buttons you press. The model becomes strangely good at this. Feed it a logged sequence and it reproduces the run almost frame for frame. Feed it a novel button sequence and it still produces plausible screens: walls in the right places, sprites in the right positions, a score counter that ticks up.

Then you ask the model to help you plan. One candidate action sequence heads straight to the door without touching the key. In the real game, the locked door blocks that route. If the model instead shows the door opening, it has reproduced the look of the game while getting a rule wrong. Later we will test a different sequence that deliberately collects the key first.

This is the central tension of tokenized game and environment models. If the representation discards the key state, fitting a larger dynamics model to those same current-state tokens cannot recover it without additional history or information. A model can reproduce what the world looks like and still be wrong about what the world does. We will make that distinction precise and then test it in a small runnable example.

Before we can talk about a model, we need to keep five things apart. In the arcade metaphor:

• The real environment is the actual game engine: it has hidden internal state (agent position, whether the key is held, the score, the frame counter) and rules that only it fully knows.
• The observation is the image of the screen that you receive. It depends on the hidden state, but usually does not reveal all of it.
• The discrete code is the symbol (or short vector of symbols) that a tokenizer assigns to that observation. It is a compressed, learnable stand-in for the frame and the dynamics model's input.
• The policy is the decision rule that maps something (frames, codes, or hidden state) to a control.
• The learned dynamics model is what we are training: a function that predicts the next code (and possibly reward and termination) from the current code plus an action.

These five objects are easy to blur together in conversation, and confusing their roles can make failures hard to diagnose. The environment owns the truth. The observation is a partial window onto that truth. The code is a learned and therefore opinionated summary of the observation. The policy is the thing that acts. The dynamics model is the thing that guesses what comes next. When a model misbehaves, ask which part of this chain is responsible.

As we saw in Part V: World-Model Architectures and again in Part II: Inference, Decisions, and Control, this chapter evaluates a world model by whether it helps an agent act. The question here is not whether the generated screen looks like the game. It is whether the learned transition law preserves the distinctions an agent needs to pick actions several steps ahead.

That distinction, between looking right and acting right, will organize what follows. A model that produces plausible open-loop video can still have a broken action interface, a lossy tokenizer, or errors on rare events. These are possible failure modes with different causes, not inevitable outcomes of tokenization. In Part VIII: Decision-Centric Research Lineages we saw how a policy can exploit errors in a model learned on narrow data. Tokenized game models can inherit that problem, so we will make the representation, data, and action interface explicit.

The plan for the chapter is:

1. Convert frames into environment tokens, and understand what that commits us to.
2. Build an action-conditioned autoregressive dynamics model over those tokens.
3. Understand what interactive generation does and does not give an agent.
4. Look at scaling and transfer across games, following a few primary papers.
5. Work through a small executable toy world that exposes the failure modes.
6. Discuss limitations, impact, and the bridge to the next chapter.

Each step builds on the one before it. Step 1 defines the symbol alphabet. Step 2 defines the probability model over that alphabet. Step 3 asks what the model is good for once it exists. Step 4 asks how far the same recipe stretches when the alphabet and the rules are shared across many games. Step 5 grounds all of it in code, so the abstractions become something you can run and poke. Step 6 draws the boundary of what this family of models can and cannot promise.

Throughout, the guiding principle is the one we set up in Part I: The World-Model Idea: a world model earns its name when it makes decisions better, not when it makes pictures prettier.

The starting point is a trajectory: a sequence of interactions with an environment. If we write the observation, action, reward, and termination flag at time $t$, the natural notation for a trajectory is

where:

• $o_t \in \mathbb{R}^{H \times W \times C}$: the rendered observation at time $t$ in the image-based setting studied here. It may be an RGB or grayscale frame, or a stack of frames, possibly with a HUD overlay. Other interfaces, such as RAM observations, require different notation.
• $a_t \in \mathcal{A}$: the discrete control chosen at time $t$. In the Arcade Learning Environment, the full discrete Atari action set has 18 controls; a game's reduced action set may contain far fewer. A multi-game dataset may use a union of the included games' actions, with each game using its applicable subset.
• $r_{t+1} \in \mathbb{R}$: the reward received after the transition into time $t+1$. Depending on the game, rewards may be sparse or frequent; pickups, scores, and deaths are possible reward events.
• $d_{t+1} \in \{0,1\}$: the termination flag for the transition into time $t+1$. When $d_{t+1}=1$, the episode ends; a subsequent reset begins a new episode.
• $T$: the length of the episode.

We use $t$ for the observation and action indices and $t+1$ for the reward index on purpose. Reward is associated with the transition after action $a_t$; in a partially observed environment it may depend on hidden state, not solely on the two rendered frames. Part II, Ch 1: Markov and Partially Observable Decision Processes used this indexing convention. A model that misaligns rewards with actions can learn the wrong decision signal even if frame reconstruction looks good.

The tuple above is a single trajectory. The index $t$ labels the observation, $t+1$ labels its reward and termination, and the episode ends at $T$, so a trajectory contains $T+1$ observations and $T$ action-reward bundles. Any model we build will be asked to predict the next element of this sequence given the preceding ones.

It is worth pausing on how much structure the raw trajectory leaves implicit. The observation $o_t$ is high-dimensional and redundant. The action $a_t$ is a small discrete label with no meaning until we connect it to an effect. The reward $r_{t+1}$ and termination $d_{t+1}$ carry explicit task feedback, yet a naive "predict the next frame" objective does not train separate predictions for them. They may also be rare compared with ordinary visual changes. The rest of this section is about the first compression step, and the following section is about the second.

A preprocessed $84 \times 84$ RGB frame contains $84 \cdot 84 \cdot 3 = 21{,}168$ channel values; this is not the native resolution of every game interface. A sixty-second clip at 30 frames per second has $30 \cdot 60 = 1{,}800$ frames. Feeding all pixel values directly into a sequence model is costly, while adjacent frames share much of their content. The model may devote capacity to predicting unchanged walls instead of small events that alter the game's state.

A tokenizer can map each observation (or short clip) into a shorter sequence of discrete symbols. If the token count is reduced by $100\times$ relative to a specified raw-pixel tokenization, the dynamics model sees roughly one-hundredth as many symbols per frame. Its predictions become class labels rather than continuous intensities. A softmax over those labels represents a categorical distribution over code outcomes; whether that distribution is calibrated must be tested separately.

The token count per frame $L$ and vocabulary size $K$ are two design knobs. $L$ is the number of codes emitted per observation; $K$ is the number of entries in the shared vocabulary. At a fixed token-count context window, increasing $L$ shortens the frame history that fits. Increasing $K$ at fixed $L$ does not change that token count, but changes the information capacity per token and the model's embedding and output-classification costs. These are separate tradeoffs, not interchangeable ways of setting the context length.

The combinatorial space of possible code sequences grows with both knobs. An autoregressive factorization is one tractable way to model a distribution over that space; it is not the only structured alternative.

The tokenizer can be one of several kinds:

• Deterministic and fixed: a hand-designed encoder, such as a color histogram or a downsample plus palette quantization.
• Deterministic and learned: a VQ-VAE or similar autoencoder whose encoder is trained to assign a specific code to each input under a nearest-neighbor rule.
• Stochastic and learned: a variational autoencoder with a discrete or relaxed-discrete bottleneck, where the encoder outputs a distribution over codes.
• Temporal: a video tokenizer that maps a short clip $(o_t, \ldots, o_{t+k})$ into a sequence of codes, so the dynamics model predicts a chunk of frames at once.

Which one you pick changes what the dynamics model sees. If the tokenizer is trained separately and frozen, the dynamics model inherits its quirks. If they are trained jointly, the dynamics objective can shape the tokenizer to keep information that matters for prediction, which is sometimes good and sometimes leads to degenerate codes. We will come back to this tradeoff.

Vector quantization is a common discrete-tokenizer design. Walking through its codebook lookup helps explain representation errors that matter later in this chapter.

Suppose an encoder $f_\phi$ maps an observation $o$ to a spatial feature map of shape $L \times D$: $f_\phi(o) \in \mathbb{R}^{L \times D}$. Think of it as a grid of $L$ vectors, each of dimension $D$. We maintain a codebook of $K$ vectors $e_1, \ldots, e_K \in \mathbb{R}^D$. For each spatial position $\ell$, we replace the feature vector with the nearest codebook entry. This is the quantize step, and it is the only place where the continuous encoder output becomes a discrete symbol:

where:

• $f_\phi(o)_\ell$: the encoder output at spatial position $\ell$, a vector in $\mathbb{R}^D$ that we want to quantize
• $e_k$: the $k$-th entry of the learned codebook, a representative vector in the same $D$-dimensional space
• $\|f_\phi(o)_\ell - e_k\|_2^2$: the squared Euclidean distance between the encoder output at position $\ell$ and codebook entry $k$, which measures how close the two vectors are
• $z_\ell$: the index of the nearest codebook entry, an integer in $\{1,\ldots,K\}$ that becomes the token for position $\ell$
• $\hat{f}_\ell$: the quantized feature at position $\ell$, obtained by looking up the nearest codebook vector $e_{z_\ell}$, so the feature map is replaced by codebook entries before decoding

The discrete codes $z = (z_1, \ldots, z_L)$ are the tokens. The decoder $g_\psi$ reconstructs an observation from the quantized features: $\hat{o} = g_\psi(\hat{f})$. Training minimizes a reconstruction loss plus a commitment term that keeps the encoder close to the codebook and a codebook term that keeps the codebook close to the encoder. The two auxiliary terms, and the straight-through estimator that lets gradients pass through the non-differentiable $\arg\min$, are engineering details; the important point for us is that the tokens are category labels, not numbers, and that the codebook is a jointly learned dictionary. A code is not "a little bit like" another code in the way two nearby real numbers are close; the codes are unordered categories, and the model has to learn any similarity structure between them from data.

The quantization step partitions the encoder's feature space by nearest codebook vector. The figure below shows the assignments; it does not depict codebook collapse, because every illustrated entry is used.

Several knobs matter for the downstream world model:

• Codebook size $K$. A small codebook can merge visually different inputs. A very large one increases the output vocabulary and may leave some codes poorly sampled; the actual usage distribution must be measured.
• Tokens per frame $L$. A larger $L$ can preserve more spatial detail, but always lengthens the dynamics sequence. Whether it retains task-relevant detail depends on the encoder and its training objective. The longer sequence also uses more of the sequence model's context window.
• Codebook collapse. During training, only a small fraction of the codebook may receive gradient, so most codes go unused. The effective vocabulary shrinks, and the tokenizer starts aliasing inputs that should have been distinct.
• Reconstruction vs. task sufficiency. The training objective rewards reconstructing the whole frame accurately. It does not care whether the reconstructed frame preserves the small detail that decides the game.

That last point deserves an example. Consider a key that occupies a $4 \times 4$ pixel region on an $84 \times 84$ screen. It covers $4 \cdot 4 = 16$ pixels out of $84 \cdot 84 = 7{,}056$, or about $0.23\%$ of the pixels. A tokenizer trained to minimize mean squared reconstruction error may devote more capacity to large, frequently visible regions such as the floor and walls than to this small key. But the game cares intensely about the key: its presence can determine whether the door opens. The tokenizer can have low average reconstruction error and still lose a variable needed for this decision. The reconstruction loss and the control task measure different things.

This is why we should not equate reconstruction error with world-model quality. As we noted in Part III, Ch 2: Sufficient State and State Abstraction, a statistic sufficient for painting need not be sufficient for decisions. A tokenizer can discard a feature needed for control while still reconstructing most pixels well. That loss can propagate downstream and may escape reconstruction-only checks; targeted event probes or control tests can reveal it earlier than a failed deployment.

A tokenized frame is an encoding that may be learned and lossy. Whether it is useful depends on the downstream task; reconstruction loss alone cannot settle whether decision-relevant distinctions survived.

With tokens in hand, the next step is to model how they change. An autoregressive sequence model conditioned on the action history is one common choice. A frame contains $L$ tokens plus reward and termination variables; a naive categorical table over all token sequences alone would have $K^L$ entries. Autoregression replaces that table with a chain of per-token conditionals. Other structured model families are possible, but this factorization gives us a precise example to analyze.

Here is one factorization, which we will use as a running example. It decomposes the joint prediction of the next frame's tokens, reward, and termination into a product over tokens within the frame, multiplied by a single conditional for the reward and termination given the completed frame:

where:

• $p_\theta(z_{t+1}, r_{t+1}, d_{t+1} \mid h_t, a_t)$: the joint distribution the model assigns to the next frame's tokens, reward, and termination flag, conditioned on the history and the current action
• $\prod_{i=1}^{L} p_\theta(z_{t+1,i} \mid h_t, a_t, z_{t+1,<i})$: the product of per-token conditionals that generates the next frame token by token, each token conditioned on the history, the action, and the tokens already emitted for that frame

Continuing the list of terms:

• $h_t$: a bounded history of previously encoded observations and actions. Depending on the model, $h_t$ might be $\{z_{t-k}, a_{t-k}, \ldots, z_t\}$ for some window $k$, or it might be a recurrent summary.
• $a_t$: the current action.
• $z_{t+1,<i}$: the tokens of the next frame that have already been generated in the autoregressive order.
• $r_{t+1}$ and $d_{t+1}$: the reward and termination flag for the transition.

Every term in this factorization deserves a comment.

The product $\prod_i p_\theta(z_{t+1,i} \mid h_t, a_t, z_{t+1,<i})$ means the model generates the next frame token by token, conditioning each prediction on the tokens already emitted for that frame. The order is a design choice: a spatial grid can be flattened in raster order, for example. The chain rule permits any fixed order to represent the same joint distribution if the conditionals are sufficiently expressive. With finite data and model capacity, however, the order can affect both training and generated behavior.

The final factor $p_\theta(r_{t+1}, d_{t+1} \mid h_t, a_t, z_{t+1})$ asks for a reward and termination flag given the generated next frame. Unlike the per-token factors, it does not need a partial next-frame prefix: all frame tokens are available. Implementations may use separate heads with their own classification or regression losses. Modeling these variables matters because a visually plausible frame with the wrong reward or termination can mislead a planner.

Training an autoregressive token model is a supervised learning problem once the tokenizer is fixed. You take a batch of trajectories, convert each observation to a token sequence, and ask the model to predict the next token sequence given the previous tokens and the observed action.

Concretely, for a training pair $(h_t, a_t, z_{t+1})$, the loss is the sum of cross-entropies over the $L$ tokens:

where:

• $\mathcal{L}_{\text{token}}(\theta)$: the token-prediction loss as a function of the model parameters $\theta$
• $p_\theta(z_{t+1,i} \mid h_t, a_t, z_{t+1,<i})$: the model's predicted probability of the $i$-th token of the next frame given the history, the action, and the already-emitted earlier tokens
• $L$: the number of tokens per frame

plus analogous loss terms for $r_{t+1}$ and $d_{t+1}$. A continuous reward may use a likelihood model or a regression loss; a small discrete reward set may use cross-entropy. Token cross-entropy is the negative log-likelihood of the observed categorical target under the model. It uses ground-truth categorical targets to train the model's conditional probabilities, but it does not eliminate the train–rollout distribution shift or make greedy, temperature-adjusted, or truncated sampling match the training distribution.

The tokens used as inputs are the actual tokens from the recorded trajectory, not the model's own samples. This is teacher forcing: we condition on ground-truth history and predict one step ahead. It is a common training regime for autoregressive models and it is what Part V, Ch 3: Autoregressive Token World Models described. Two implications matter for evaluation:

1. It trains the model to be accurate on data distribution histories, not on its own histories. During a planning rollout, small errors can compound and move the model's inputs away from the training distribution.
2. It supplies ground-truth $h_t$ during training. A model may exploit cues in that history that its own rollouts later fail to preserve; teacher forcing does not itself require such reliance, and $h_t,a_t$ need not determine a stochastic next frame.

The second point is subtle. Suppose the next frame depends on whether the agent holds the key, and that fact is visible in $h_t$ because the key sprite has disappeared. The model may predict observed next frames accurately by using that cue. Such accuracy alone does not show that its own generated histories will preserve the cue or that it will handle a new pickup sequence. During rollout, an error in the generated history can remove the information on which later predictions depend. That is one way strong one-step accuracy can coexist with weak multi-step simulation; it is not proof that every teacher-forced model failed to learn the rule.

Causal masking enforces this autoregressive structure inside a transformer: token $i$ attends only to tokens $j < i$ and to the context $h_t, a_t$. If training exposed later target tokens to that predictor, the stated next-token likelihood would leak information. The mask prevents that leakage; other generative architectures can enforce a valid factorization in different ways.

Once trained, we generate a future by sampling from the factorized distribution. Which sampling mode we choose changes the distribution of the futures we observe, and it changes how the model behaves inside a planner. There are several sampling modes:

• Greedy: take the argmax at each step. Deterministic but can produce degenerate, repetitive outputs when the model is uncertain.
• Temperature sampling: sample from $\text{softmax}(\text{logits} / \tau)$, where $\tau > 0$ is the temperature parameter: a scalar that scales the logits before the softmax is applied. Small $\tau$ approaches greedy; large $\tau$ flattens the distribution. Temperature changes concentration and diversity; its effect on fidelity depends on the task, metric, and calibration.
• Top-$k$ or nucleus (top-$p$) sampling: when their thresholds exclude tokens, restrict sampling to a high-probability subset. This can remove rare but decision-relevant events such as a pickup or death.

Sampling mode changes the futures a planner considers. Greedy decoding gives one model future. Repeated temperature samples expose several model futures at additional cost, but they are only useful if the model distribution is accurate and calibrated. A top-$k$ or top-$p$ threshold that excludes a low-probability pickup or death removes that event from sampled plans, biasing the evaluation of those plans.

When the environment is stochastic, the same conditioned history and action can have several valid next outcomes. The observed sample still supplies a valid negative-log-likelihood training target; across repeated samples, cross-entropy can learn the conditional distribution. A single point prediction or an overconfident distribution would instead misrepresent outcome variation, such as random enemy spawns or item drops.

Token prediction alone supplies code labels, not explicit reward or termination predictions. If a known reward or stopping rule can be evaluated from the predicted trajectory, including its actions and transitions where relevant, it can provide those signals; otherwise the model needs learned heads or another estimator. A decoded picture may look like the game, but visual fidelity alone does not tell a planner the return or remaining horizon.

A planner cannot reliably score candidate trajectories by return without a reliable reward mapping for its predicted trajectories, whether learned or known. Without a reliable termination signal, a model rollout may continue past an event that ends the real episode. Frame reconstruction alone does not test either signal.

Game trajectories have several temporal conventions that a dynamics model must respect.

• Action repeat. Many agents operate on a frame-level control rate of 60 Hz, but the agent chooses an action every $k$ frames, and the environment executes that action for the next $k$ consecutive frames. The value of $k$ becomes part of the effective time step of the model: one action corresponds to $k$ consecutive raw frames. If you forget this in the data pipeline, the model learns to predict an action's effect over an inconsistent time horizon.
• No-op controls. Some datasets include a "do nothing" action, sometimes as a legal action and sometimes as a technique for skipping events. The model treats it as just another action, so it must see it in the data.
• Frame skip and image stacking. The observation is often a stack of $k$ frames or a max-pooled pair, so the model's input is not a single image. The dynamics model learns a different causal structure than it would from single frames.
• Control latency. In some games and many robots, the action takes effect one or more steps after being issued. A model that assumes zero latency will systematically mis-predict transitions where the true effect lags.

These conventions are common in benchmark pipelines, though their exact settings vary. The preprocessing pipeline is part of the model's action interface. If an action means "press fire for four frames" during training but "press fire for one frame" at test time, the learned conditional transition no longer matches the queried process. The discrepancy can appear as systematic prediction error even when the model's reported uncertainty is low.

There are two scales at which you can train a tokenized game model. A single-game model is trained on trajectories from one environment. A multi-game model is trained on a union of trajectories from many environments, sometimes with an explicit game identifier and sometimes with environment-specific action adapters. The two regimes face different problems.

For a single game, $a_t$ has a fixed meaning within the chosen action interface, and the tokenizer and dynamics can be fitted to that game's data. Its tokens may specialize to the game's art style. The codebook size, training cost, and accuracy still depend on the architecture, data, and task; specialization alone guarantees none of them.

For multi-game, several complications arise:

• Observation representations must be compatible with the shared dynamics model. Differing resolutions, aspect ratios, or palettes may be handled by a shared tokenizer with preprocessing, or by game-specific encoders or adapters that produce compatible model inputs. Downsampling and style normalization are options, not requirements.
• Game identity or context is required. Because the dynamics depend on the game rules, the model must condition on something that identifies the game. This can be a one-hot game embedding, a learned "environment ID" token prepended to the sequence, or natural language. Text-prompt conditioning is taken up in Chapter 53 on Cosmos and omnimodal world foundation models, in Part IX: Foundation and World-Action Models.
• Action sets differ. A shared integer ID for "button 3" need not mean the same underlying effect in every game. "Jump" in one platformer might be "dash" in another and "no-op" in a third. Action masking can constrain the legal choices; per-game adapters or game-conditioned shared dynamics can model differing effects. The interface should state which semantics are shared and which depend on context.

The scaling opportunity is real: a shared tokenizer sees more visual variety, and a shared dynamics model sees more rule variety. Whether that leads to transfer depends on how well the model can condition on game identity and reuse behavior across games. It is a scaling hypothesis, not a guarantee. A model that shares a tokenizer but not a dynamics model can reuse visual features without necessarily transferring rules. A model that shares both needs enough game context to avoid mixing incompatible transition laws.

A tokenized model can be interactive, meaning you can condition on an action and read a sample from the model, but "interactive" is not the same as "useful for control". The distinctions matter enough to spell out.

The basic loop is:

1. Initialize. Provide an initial observation or a history of $k$ observations.
2. Choose an action. A policy, a human, or a planner picks $a_t$.
3. Sample the model. Generate $\hat{z}_{t+1}, \hat{r}_{t+1}, \hat{d}_{t+1}$ from the model.
4. Decode. If a display is needed, run $\hat{z}_{t+1}$ through the decoder. If not (for internal use), keep it as tokens.
5. Advance. Append $\hat{z}_{t+1}$ and the chosen action to the history and go to step 2.

The loop can serve as a simulated environment, but its state is whatever the model and its context retain, not the game's inaccessible ground-truth state. A recurrent model may carry its own hidden memory; a history-based model may carry past tokens and actions. Neither guarantees that a decision-relevant variable discarded from the observations can be recovered. Different samples can also produce different futures from the same history, which is expected for a stochastic model but must be evaluated for consistency with the real environment.

Interactive token models are often used for three distinct purposes, and each has its own validation metric.

• Playable generated environments. A human sits at the keyboard and plays the model as if it were a game. The validation is whether the experience is engaging, coherent, and controllable. Chapter 52 on Genie and interactive generated environments develops this framing in Part IX: Foundation and World-Action Models.
• Model rollouts for planning or policy learning. A decision-making algorithm plans over $p_\theta$ and returns an action sequence. The validation is whether acting on that sequence in the real environment produces the reward the planner expected.
• Conditional video demonstrations. The model is used to synthesize demonstration-like sequences for training or for entertainment. The validation here is a bit softer: fidelity to the style and content of real demonstrations.

The three uses share a plumbing, but their validation targets differ. A model that is a great playable environment may be a bad planner. A model that produces excellent demonstrations may not be resampled in a way that lets a planner exploit it. Conflating the three is a common mistake, because all three look like "the model generates the game", but only the middle one is a claim about decision quality.

Imagine optimizing a policy inside the learned model. Optimization can favor action histories where model error makes imagined reward look better than real reward. This is model exploitation, discussed in Part VIII, Ch 6: Offline and Conservative Model-Based RL. A token model may still assign a sharp predictive distribution to a poorly supported history; sharpness alone is not evidence that the prediction is accurate there.

Suppose the training data contains individual moves but never the complete key-to-door sequence. A rollout must predict observations after the pickup under a history with little direct support. The model will still return a distribution, but that distribution may be wrong or poorly calibrated. An optimizing policy can exploit optimistic errors if no evaluation or constraint catches them.

One-step held-out likelihood is not enough to catch this. The model can have excellent next-token accuracy on logged steps while mis-modeling a novel action sequence. Fixed-action branch rollouts test multi-step trajectory fidelity; feedback-controlled evaluations go further by testing how a policy reacts to the states the model or real environment actually produces.

The three evaluation regimes we should keep distinct are:

The closed-loop test is the most direct measure of whether a model supports the intended policy, and it requires access to the real environment. Fixed-action branches are cheaper diagnostics of the learned transition law, but they do not measure an adaptive policy's return.

A useful fixed-action branch protocol is to pick a starting context, choose an action sequence different from the logged one, and run both the model and the real environment under that same sequence. Compare event frequencies, reward, termination, and state summaries at several branch points. This probes counterfactual transitions, but it remains open-loop because later actions do not respond to the generated observations. For a closed-loop test, run a specified policy in each system and let it choose each next action from the feedback it actually receives.

If the environment is stochastic, do not compare pixel-by-pixel to a single realized future. Instead, compare the distribution of futures under the model to the distribution under the real environment. This requires sampling multiple rollouts from the same context and comparing summaries: event frequencies, return distributions, termination rates. Pixel-matching against one sample can count ordinary stochastic variation as model error; one pair of samples cannot establish whether the predicted distribution is wrong.

The two failure modes are distinguishable if you look carefully:

• Aleatoric diversity: the model emits multiple plausible futures under the same action, and the real environment would also emit multiple plausible futures. This is healthy.
• Epistemic uncertainty about rarely observed actions: limited data leave the transition law poorly known. A model may nonetheless emit a sharp and wrong distribution, which can mislead a planner.

A single predictive distribution from one fitted model does not, by itself, separate outcome randomness from uncertainty about the learned transition law. Models with explicit parameter-uncertainty estimates, such as suitable ensembles or Bayesian approximations, can attempt that separation. Calibration and out-of-distribution checks are also relevant when planning decisions must be reliable; Part XII: Reliable World Models develops those tests.

Because the interactive loop feeds the model's own samples back in, small per-step errors can compound. After $H$ steps, where $H$ is the rollout horizon, the model's input distribution may differ from the data distribution. The rate at which this drift grows depends on the model's error, the degree of stochasticity, the degree of control, and the presence of absorbing events. There is no single number that characterizes it. Tight control, randomness, and feedback can change how errors propagate, but none alone determines whether a particular rollout drifts slowly or quickly.

What matters depends on the task and model. Per-step prediction errors can accumulate even over short horizons. Over longer horizons, missed events such as pickups, deaths, or terminations may dominate the decision error even when the pixels remain plausible. A planner's useful horizon must therefore be tested against the task and controller; Part VII, Ch 1: Sampling-Based Planning and Model Predictive Control describes short-horizon replanning as one way to limit exposure to model drift.

An interactive model that is spectacular at five steps and garbage at fifty can still be useful for short-horizon control and useless for planning to a distant goal. This is why the horizon must be treated as a property of the system (model plus controller plus task), not of the model alone. The same model can be entirely adequate for one planning problem and entirely inadequate for another.

The horizon at which a model becomes unusable depends on the model, controller, and task together. For illustration only, assume each step has an independent error event with the same probability $\epsilon$. The probability of at least one error by step $H$ is $1-(1-\epsilon)^H$. Real rollout errors need not be independent or constant-rate, so the curves below are not empirical predictions for a particular world model.

A small single-game model is already interesting. The bigger story, and the one that has driven much of the recent literature, is training on many games and hoping that the resulting tokens, dynamics, and interfaces transfer.

Rather than survey the literature broadly, we will look closely at three primary papers that illustrate distinct design choices. We deliberately do not quote leaderboard numbers: reported scores can depend materially on environment versions, action-set conventions, evaluation frames, and data budgets, so direct comparison requires matched protocols.

IRIS: Transformers are Sample-Efficient World Models. Micheli, Alonso, and Fleuret (2022, arXiv:2209.00588) combine a discrete autoencoder with an autoregressive Transformer world model. Its codes are the world model's prediction space, while its actor-critic is trained using imagined rollouts presented as decoded images. IRIS illustrates a single-game regime with per-game representation and dynamics models. The reported decision outcome is sample efficiency in the real game, distinct from video fidelity.

JOWA: Scaling Offline Model-Based RL via Jointly-Optimized World-Action Model Pretraining. Introduced by Cheng et al. (2024, arXiv:2410.00564), JOWA pretrains a world-action model on multi-game offline data and studies transfer. Its joint optimization refers to the shared Transformer backbone trained with world-model and action-value objectives. The paper trains a VQ-VAE tokenizer in an earlier stage and freezes it during joint world-action training; it does not show that the tokenizer itself becomes decision-sufficient through that stage. The transfer question is what gets adapted for a new game, using how much target-game data and interaction.

DART: Learning to Play Atari in a World of Tokens. Introduced by Agarwal, Andrews, and Ebrahimi Kahou (2024, arXiv:2406.01361), DART uses discrete representations, an autoregressive Transformer-decoder world model, and a Transformer-encoder behavior model. It is a contrast to IRIS and JOWA in how tokenized predictions are paired with behavior learning. These papers share a broad tokenized-world-model pattern while differing in their training regimes and policy components.

Genie: Generative Interactive Environments (Bruce et al., 2024, Proceedings of the 41st International Conference on Machine Learning) is a forward pointer rather than a peer. Its video tokenizer and autoregressive dynamics are directly relevant to this chapter, but its most distinctive contribution is the inference of latent actions from unlabeled video, the topic of Chapter 52 on Genie and interactive generated environments in Part IX: Foundation and World-Action Models. We mention it here only because it belongs to the same family and because the same questions about action interfaces and closed-loop evaluation apply.

Notice what we have not claimed: we have not claimed that any of these models transfer zero-shot to new games, that any of them solve long-horizon planning, or that they are the best models in their respective benchmarks. We have simply named a few primary sources for the broad design pattern.

The phrase "the model transfers across games" hides several distinct mechanisms. Each one can fail independently.

• Reusable visual codes. A tokenizer trained on many games may reuse visual features such as edges or sprite shapes. Whether this helps on a new game depends on its imagery and on whether those features preserve task-relevant details; measure visual transfer separately.
• Shared dynamics priors. The model may reuse patterns such as continued motion or contact followed by collection. This may help when games share transition rules, but transfer can fail when their rules differ, as between a projectile shooter and a turn-based puzzle.
• Transferable action semantics. Some controls, such as "up", may have aligned effects across games, while a shared action ID need not mean the same thing everywhere. Overlapping controls can transfer; ambiguous or game-specific ones require context, masking, or adaptation. A latent action alphabet or learned action embedding is another possible interface.
• A shared decision head. The policy or value head trained alongside the model can transfer if the semantics of "good" and "bad" are sufficiently similar across games. This is fragile: a decision head that plays Pac-Man well is not obviously a good head for a racing game.

Each of these can succeed while the others fail, so an ablation study should isolate them. If the visual codes help but the dynamics priors do not, you want to know, because the fix is different in the two cases. A transfer result that reports a single aggregate number hides which mechanism produced the gain.

Different transfer mechanisms rely on different kinds of shared structure. The figure below uses invented scores to illustrate hypotheses one might test as game similarity changes. It is not evidence that one mechanism transfers better than another.

The following protocol makes the transfer claim testable.

1. Hold out environments, not just levels. Reserve entire games from pretraining. Holding out only levels tests intra-game generalization, a different question from cross-game transfer.
2. Specify what is adapted at test time. Does the model fine-tune its tokenizer? Its dynamics? A decision head? An action adapter? Each option defines a different question.
3. Specify adaptation data and evaluation interaction separately. A zero-shot transfer claim normally excludes target-game training or parameter updates, but a controller still interacts with the game during evaluation. Few-shot adaptation uses a stated small target-game dataset. In-context adaptation uses target-game context without weight updates. Report both the adaptation budget and the interactions used to score the resulting controller.
4. Compare against a from-scratch baseline. Training a model on the new game alone helps show whether pretraining produced a transfer gain under the same evaluation budget.
5. Include a no-pretraining ablation. Even a pretrained tokenizer can dominate the effect size; without an ablation you cannot tell whether pretraining the dynamics mattered.
6. Distinguish three transfer questions. Transfer within a game (new levels) is different from transfer to a new game with the same action set, which is different from zero-shot control on a new game with a different action set. Solving one does not imply solving the others.

The following is not a rule, but a rule of thumb: if a cross-game transfer result does not specify the action interface, the data budget, and the level of adaptation, it is not interpretable.

Now that the concepts are on the table, let us build a minimal example that exercises them. We will use a $5 \times 5$ tile world with an agent, a key, a locked goal, a wall, and a hazard. It is small enough to reason about completely and rich enough to exhibit the failures we care about.

The story we will demonstrate:

1. A discrete tokenizer that preserves the key sprite yields a model that behaves correctly on a plan that goes to the key first.
2. A lossy tokenizer that aliases the key sprite into "empty" yields a model with similar one-step accuracy but a broken multi-step plan: the model cannot tell whether the agent has the key, so it cannot open the door.
3. A restricted dataset that never includes key pickups yields a model with no in-support entry for that transition. Our rollout applies an explicit stay-put fallback, rather than estimating a confident prediction.

The dynamics model is an empirical count table over observed transitions: for each $(\text{obs}, a)$ pair, we store next-observation outcomes with reward and termination sums. It is a pedagogical analogue, not an autoregressive Transformer or a smoothed estimator. The toy isolates representation and data support without large-model training. A more expressive model might use action history to recover some information hidden from the current lossy observation, but changing the model class alone does not guarantee that it will learn the missing pickup rule or handle an unobserved action branch.

We start by defining the grid, the tile alphabet, and the transition function.

The picture shows the layout that the tokenizer must encode. What we will manipulate next is not the picture but the tokenization scheme that reads it, because the scheme decides which of these tiles survive into the model's input.

The full observation is the integer grid. The lossy observation replaces every KEY tile with EMPTY. This mimics a tokenizer that assigns no code to the key sprite, either because the codebook was trained with a small $K$ or because the reconstruction loss did not care about a tiny detail.

For reachable states in this toy world, the full frame reveals whether the agent holds the key: the key sprite disappears after pickup. The lossy tokenizer removes that sprite in both cases. At the same apparent position, a key-held and a key-unheld state can therefore produce the same lossy observation but different outcomes for an action toward the goal. The lossy current observation is not Markov-sufficient; an action history that records the pickup could restore the distinction.

We generate trajectories under a uniform exploration policy and record $(o_t, a_t, o_{t+1}, r_{t+1}, d_{t+1})$ tuples.

The restricted policy still visits most of the grid. What it avoids is the single transition into KEY_POS.

The model is a lookup table. For each observed $(\text{obs}, a)$ pair we store the empirical distribution over next observations and the mean reward and termination under that outcome.

Before we test the models on multi-step plans, let us look at the metric that is easy to compute and misleading: held-out next-token accuracy.

The full and lossy models score comparably here, but the lossy model is evaluated in its own token space, where the key sprite has been removed. The restricted model's perfect score is more selective: its denominator excludes 6,163 of the 10,393 held-out queries because it has no table entry for them. Accuracy must be read alongside coverage, and neither number alone tells us how the planned key-to-door sequence will unfold.

We hand the model a plan that goes down to the key, picks it up, walks to the goal, and stops. The same plan is executed in the true environment, in the full model, in the lossy model, and in the restricted model. The four resulting distance-to-goal curves reveal which models preserve the pickup rule and which ones stall short of the goal.

The chart is a fixed-action branch test. It does not score picture similarity or next-token likelihood; it compares the predicted outcome of one specified plan with the environment's outcome. Two models miss the goal despite high conditional one-step accuracy. This is not a closed-loop test of a policy choosing new actions from feedback.

The restricted model deserves a closer look. It was trained on thousands of trajectories from the same grid, so it knows the world quite well, except that it never saw a key held. We can measure the effect crisply: what fraction of the key-held observations in the held-out test set does the restricted model have an entry for?

Both impaired models predict the wrong outcome for the fixed plan, but this code does not measure their confidence. The lossy model chooses a frequent next observation from aliased data. The restricted model has no entry for the pickup action; predict_next returns None, and rollout_model substitutes the unchanged observation. That fallback is deliberately simple, not a general safety guarantee. A neural autoregressive model normally emits a distribution even for a poorly supported query, so its uncertainty and calibration need separate tests.

The toy failure is benign, but the diagnostic carries over: inspect whether a proposed action branch lies within the model's data support, and test its predicted consequences against the real environment before trusting it for control.

Tokenized game models use compact discrete observation sequences and supervised predictive losses, but their reachable code space can still be large and autoregressive sampling can be costly. Their limitations arise from representation, data support, action semantics, and the way a planner uses their predictions.

The first limitation is the tokenizer. If the current code and available history contain no trace of a decision-relevant variable, the dynamics model cannot condition on it. In our toy, the count model sees only the current lossy grid, so removing the key tile aliases key-held and key-unheld states. Reconstruction-oriented objectives, especially a uniform pixel loss, can underweight small task-relevant details relative to large surfaces. A task-aware loss, longer action history, or a separate symbolic channel may help, but each needs its own test of whether the required distinction survives.

The second limitation is the action interface. A shared token model needs to interpret the action it receives. Even in one game, latency, action repeat, and variable frame rates can create a mismatch; multiple games add differing controls. Cross-game transfer results therefore need to state what is shared and what is adapted at test time. A policy trained inside a token model can earn high imagined returns yet perform poorly in the real game when it exploits an action-interface or transition-model error.

The third limitation is data support. When a model is trained offline on trajectories, its reliable coverage depends on those trajectories. Expert demonstrations may concentrate on successful paths and omit alternative actions; exploratory data may visit different states without reaching a task objective. Neither collection strategy guarantees coverage of the compound action sequences that a planner wants to evaluate. Our toy world exhibits this cleanly: the restricted dataset covers most of the grid but completely misses the pickup transition, so the model has no entry for the state it needs. Real datasets are messier: coverage is partial, action distributions are uneven, and rare events are rare for a reason. You should not assume that a large dataset implies coverage of the specific branch you care about. This is the same concern we discussed in Part VIII, Ch 6: Offline and Conservative Model-Based RL, where the mismatch between the data distribution and the query distribution motivates conservative evaluation.

The fourth limitation is evaluation. Images are easy to inspect, but a visually coherent rollout can hide a wrong reward, a missed termination, or an inconsistent level layout. No single scalar certifies a world model: reconstruction, token likelihood, event prediction, reward, action sensitivity, calibration, long-horizon consistency, and real closed-loop return answer different questions. A result reported on one metric should not be read as a result on the others.

Tokenized game models offer a useful testbed. Existing game trajectories can train predictive models, and discrete codes can reduce sequence length compared with raw pixel tokenization. Resettable environments support repeatable policy-return tests; comparing counterfactual branches from the same interior context additionally needs snapshot/restore, reproducible replay, or equivalent state control. Sampling cost still depends on model size, token count, and decoding scheme, so cheap imagined trajectories are not guaranteed. These systems let us test ideas from Part VII: Planning and Agency and Part VI: Learning World Models at Scale under controlled conditions.

The practical implication is that a tokenized game model should be described with the same care as any other world model. Which data was it trained on, with which behavior policy, over which action interface, and with what testing regime? Under what conditions does it transfer, and under what conditions does it fail? And, most importantly, what evidence do we have that the learned transition law preserves the distinctions an agent needs, rather than merely reproducing the appearance of the world?

This is the setup for the next chapter. Our toy world used a hand-designed tokenizer: the grid of tile types is already discrete, and we only had to decide whether to include the key. Real environments are not so kind. Chapter 50, Pretrained Visual Models and DINO-WM, in Part IX: Foundation and World-Action Models, changes the representation choice fundamentally: it keeps a continuous feature space learned by a pretrained visual model, and it predicts in that space instead of predicting discrete codes. The intuition we built here, that a representation is only as useful as the decisions it supports, carries over directly. The mechanisms do not.

Tokenized game and environment models encode observations as sequences of discrete symbols. An observation tokenizer $E$ maps frames to $z_t \in \{1,\ldots,K\}^L$; one possible action-conditioned model autoregressively predicts the next tokens, reward, and termination. Tokenization can shorten the sequence relative to a specified raw-pixel representation, while the cost of sampling and planning still depends on $K$, $L$, and the dynamics architecture.

The chapter's key points are summarized below.

• Tokenization is a representation choice. The tokenizer $E$ is not a neutral preprocessing step. It decides which features the dynamics model can reason about, and it can have low reconstruction error while discarding the variable that determines whether the environment is solvable.
• The autoregressive factorization is a design template. A discrete token model with an action-conditioned prior, plus reward and termination heads, is one convenient structure. Alternatives exist and are legitimate, and different choices trade capacity, training stability, and sampling speed.
• Time alignment is part of the model. Action repeat, no-op controls, stacking, and control latency affect the transition data. If timing information needed for prediction is absent from the model inputs, increasing model size alone cannot supply it.
• Interactive generation is not the same as decision usefulness. A model can be a great playable environment and a bad planner, or vice versa. Three uses demand three validation targets: playability, planning value, and demonstration quality.
• Separate branch tests from closed-loop tests. One-step accuracy does not certify a world model. Fixed-action branches test multi-step transition fidelity; adaptive policy runs in the real environment test control usefulness.
• Data support and out-of-distribution queries matter. A neural token model may emit a prediction even on poorly supported inputs. Coverage, calibration, uncertainty checks, or conservative planning can help determine whether a proposed plan should be trusted.
• Transfer has separable components. Visual codes, dynamics priors, action semantics, and decision heads can succeed or fail independently. Cross-game results should specify which mechanism is tested, how the model is adapted, and what baselines are used.

The next chapter takes the representation question one step further. Instead of training a discrete tokenizer and an autoregressive dynamics model over its codes, we will look at models that keep a continuous feature space from a frozen pretrained visual backbone and predict in that space. The same core question applies: does the learned transition preserve the distinctions the agent needs, or does it only preserve the ones that make the video look right?

The key parameters for the count-based toy dynamics model are:

• Tokenizer: the function that maps each observation to its discrete code. Whether the key sprite is preserved determines whether the model can reason about the pickup rule.
• Codebook size $K$: the number of distinct code symbols. A small $K$ can encourage aliasing; a large $K$ may leave some codes poorly sampled. Measure actual code usage and task-relevant distinctions rather than inferring either outcome from $K$ alone.
• Tokens per frame $L$: the number of discrete codes emitted per observation. Larger $L$ can preserve more spatial detail, depending on the tokenizer, but it always lengthens the dynamics sequence and uses more context-window capacity.
• Training data support: the coverage of the training trajectories. A missing pickup transition leaves the count model without an entry; a neural model may produce a prediction, but its confidence and accuracy on that branch require separate assessment.
• Temporal and action interface: frame stacking determines the observation history, while action repeat, no-op controls, and control latency determine how controls align with transitions. A mismatch in either can make the learned dynamics wrong.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about tokenized game and environment models.