Tokenized Game and Environment Models › Setting up the question

Before we can talk about a model, we need to keep five things apart. In the arcade metaphor:

• The real environment is the actual game engine: it has hidden internal state (agent position, whether the key is held, the score, the frame counter) and rules that only it fully knows.
• The observation is the image of the screen that you receive. It depends on the hidden state, but usually does not reveal all of it.
• The discrete code is the symbol (or short vector of symbols) that a tokenizer assigns to that observation. It is a compressed, learnable stand-in for the frame and the dynamics model's input.
• The policy is the decision rule that maps something (frames, codes, or hidden state) to a control.
• The learned dynamics model is what we are training: a function that predicts the next code (and possibly reward and termination) from the current code plus an action.

These five objects are easy to blur together in conversation, and confusing their roles can make failures hard to diagnose. The environment owns the truth. The observation is a partial window onto that truth. The code is a learned and therefore opinionated summary of the observation. The policy is the thing that acts. The dynamics model is the thing that guesses what comes next. When a model misbehaves, ask which part of this chain is responsible.

As we saw in Part V: World-Model Architectures and again in Part II: Inference, Decisions, and Control, this chapter evaluates a world model by whether it helps an agent act. The question here is not whether the generated screen looks like the game. It is whether the learned transition law preserves the distinctions an agent needs to pick actions several steps ahead.

That distinction, between looking right and acting right, will organize what follows. A model that produces plausible open-loop video can still have a broken action interface, a lossy tokenizer, or errors on rare events. These are possible failure modes with different causes, not inevitable outcomes of tokenization. In Part VIII: Decision-Centric Research Lineages we saw how a policy can exploit errors in a model learned on narrow data. Tokenized game models can inherit that problem, so we will make the representation, data, and action interface explicit.

The plan for the chapter is:

1. Convert frames into environment tokens, and understand what that commits us to.
2. Build an action-conditioned autoregressive dynamics model over those tokens.
3. Understand what interactive generation does and does not give an agent.
4. Look at scaling and transfer across games, following a few primary papers.
5. Work through a small executable toy world that exposes the failure modes.
6. Discuss limitations, impact, and the bridge to the next chapter.

Each step builds on the one before it. Step 1 defines the symbol alphabet. Step 2 defines the probability model over that alphabet. Step 3 asks what the model is good for once it exists. Step 4 asks how far the same recipe stretches when the alphabet and the rules are shared across many games. Step 5 grounds all of it in code, so the abstractions become something you can run and poke. Step 6 draws the boundary of what this family of models can and cannot promise.

Throughout, the guiding principle is the one we set up in Part I: The World-Model Idea: a world model earns its name when it makes decisions better, not when it makes pictures prettier.