Tokenized Game and Environment Models › What interactive generation does and does not give an agent

A tokenized model can be interactive, meaning you can condition on an action and read a sample from the model, but "interactive" is not the same as "useful for control". The distinctions matter enough to spell out.

The basic loop is:

1. Initialize. Provide an initial observation or a history of $k$ observations.
2. Choose an action. A policy, a human, or a planner picks $a_t$.
3. Sample the model. Generate $\hat{z}_{t+1}, \hat{r}_{t+1}, \hat{d}_{t+1}$ from the model.
4. Decode. If a display is needed, run $\hat{z}_{t+1}$ through the decoder. If not (for internal use), keep it as tokens.
5. Advance. Append $\hat{z}_{t+1}$ and the chosen action to the history and go to step 2.

The loop can serve as a simulated environment, but its state is whatever the model and its context retain, not the game's inaccessible ground-truth state. A recurrent model may carry its own hidden memory; a history-based model may carry past tokens and actions. Neither guarantees that a decision-relevant variable discarded from the observations can be recovered. Different samples can also produce different futures from the same history, which is expected for a stochastic model but must be evaluated for consistency with the real environment.

Interactive token models are often used for three distinct purposes, and each has its own validation metric.

• Playable generated environments. A human sits at the keyboard and plays the model as if it were a game. The validation is whether the experience is engaging, coherent, and controllable. Chapter 52 on Genie and interactive generated environments develops this framing in Part IX: Foundation and World-Action Models.
• Model rollouts for planning or policy learning. A decision-making algorithm plans over $p_\theta$ and returns an action sequence. The validation is whether acting on that sequence in the real environment produces the reward the planner expected.
• Conditional video demonstrations. The model is used to synthesize demonstration-like sequences for training or for entertainment. The validation here is a bit softer: fidelity to the style and content of real demonstrations.

The three uses share a plumbing, but their validation targets differ. A model that is a great playable environment may be a bad planner. A model that produces excellent demonstrations may not be resampled in a way that lets a planner exploit it. Conflating the three is a common mistake, because all three look like "the model generates the game", but only the middle one is a claim about decision quality.

Imagine optimizing a policy inside the learned model. Optimization can favor action histories where model error makes imagined reward look better than real reward. This is model exploitation, discussed in Part VIII, Ch 6: Offline and Conservative Model-Based RL. A token model may still assign a sharp predictive distribution to a poorly supported history; sharpness alone is not evidence that the prediction is accurate there.

Suppose the training data contains individual moves but never the complete key-to-door sequence. A rollout must predict observations after the pickup under a history with little direct support. The model will still return a distribution, but that distribution may be wrong or poorly calibrated. An optimizing policy can exploit optimistic errors if no evaluation or constraint catches them.

One-step held-out likelihood is not enough to catch this. The model can have excellent next-token accuracy on logged steps while mis-modeling a novel action sequence. Fixed-action branch rollouts test multi-step trajectory fidelity; feedback-controlled evaluations go further by testing how a policy reacts to the states the model or real environment actually produces.

The three evaluation regimes we should keep distinct are:

The closed-loop test is the most direct measure of whether a model supports the intended policy, and it requires access to the real environment. Fixed-action branches are cheaper diagnostics of the learned transition law, but they do not measure an adaptive policy's return.

A useful fixed-action branch protocol is to pick a starting context, choose an action sequence different from the logged one, and run both the model and the real environment under that same sequence. Compare event frequencies, reward, termination, and state summaries at several branch points. This probes counterfactual transitions, but it remains open-loop because later actions do not respond to the generated observations. For a closed-loop test, run a specified policy in each system and let it choose each next action from the feedback it actually receives.

If the environment is stochastic, do not compare pixel-by-pixel to a single realized future. Instead, compare the distribution of futures under the model to the distribution under the real environment. This requires sampling multiple rollouts from the same context and comparing summaries: event frequencies, return distributions, termination rates. Pixel-matching against one sample can count ordinary stochastic variation as model error; one pair of samples cannot establish whether the predicted distribution is wrong.

The two failure modes are distinguishable if you look carefully:

• Aleatoric diversity: the model emits multiple plausible futures under the same action, and the real environment would also emit multiple plausible futures. This is healthy.
• Epistemic uncertainty about rarely observed actions: limited data leave the transition law poorly known. A model may nonetheless emit a sharp and wrong distribution, which can mislead a planner.

A single predictive distribution from one fitted model does not, by itself, separate outcome randomness from uncertainty about the learned transition law. Models with explicit parameter-uncertainty estimates, such as suitable ensembles or Bayesian approximations, can attempt that separation. Calibration and out-of-distribution checks are also relevant when planning decisions must be reliable; Part XII: Reliable World Models develops those tests.

Because the interactive loop feeds the model's own samples back in, small per-step errors can compound. After $H$ steps, where $H$ is the rollout horizon, the model's input distribution may differ from the data distribution. The rate at which this drift grows depends on the model's error, the degree of stochasticity, the degree of control, and the presence of absorbing events. There is no single number that characterizes it. Tight control, randomness, and feedback can change how errors propagate, but none alone determines whether a particular rollout drifts slowly or quickly.

What matters depends on the task and model. Per-step prediction errors can accumulate even over short horizons. Over longer horizons, missed events such as pickups, deaths, or terminations may dominate the decision error even when the pixels remain plausible. A planner's useful horizon must therefore be tested against the task and controller; Part VII, Ch 1: Sampling-Based Planning and Model Predictive Control describes short-horizon replanning as one way to limit exposure to model drift.

An interactive model that is spectacular at five steps and garbage at fifty can still be useful for short-horizon control and useless for planning to a distant goal. This is why the horizon must be treated as a property of the system (model plus controller plus task), not of the model alone. The same model can be entirely adequate for one planning problem and entirely inadequate for another.

The horizon at which a model becomes unusable depends on the model, controller, and task together. For illustration only, assume each step has an independent error event with the same probability $\epsilon$. The probability of at least one error by step $H$ is $1-(1-\epsilon)^H$. Real rollout errors need not be independent or constant-rate, so the curves below are not empirical predictions for a particular world model.