Tokenized Game and Environment Models › Scaling and transfer across games

A small single-game model is already interesting. The bigger story, and the one that has driven much of the recent literature, is training on many games and hoping that the resulting tokens, dynamics, and interfaces transfer.

Rather than survey the literature broadly, we will look closely at three primary papers that illustrate distinct design choices. We deliberately do not quote leaderboard numbers: reported scores can depend materially on environment versions, action-set conventions, evaluation frames, and data budgets, so direct comparison requires matched protocols.

IRIS: Transformers are Sample-Efficient World Models. Micheli, Alonso, and Fleuret (2022, arXiv:2209.00588) combine a discrete autoencoder with an autoregressive Transformer world model. Its codes are the world model's prediction space, while its actor-critic is trained using imagined rollouts presented as decoded images. IRIS illustrates a single-game regime with per-game representation and dynamics models. The reported decision outcome is sample efficiency in the real game, distinct from video fidelity.

JOWA: Scaling Offline Model-Based RL via Jointly-Optimized World-Action Model Pretraining. Introduced by Cheng et al. (2024, arXiv:2410.00564), JOWA pretrains a world-action model on multi-game offline data and studies transfer. Its joint optimization refers to the shared Transformer backbone trained with world-model and action-value objectives. The paper trains a VQ-VAE tokenizer in an earlier stage and freezes it during joint world-action training; it does not show that the tokenizer itself becomes decision-sufficient through that stage. The transfer question is what gets adapted for a new game, using how much target-game data and interaction.

DART: Learning to Play Atari in a World of Tokens. Introduced by Agarwal, Andrews, and Ebrahimi Kahou (2024, arXiv:2406.01361), DART uses discrete representations, an autoregressive Transformer-decoder world model, and a Transformer-encoder behavior model. It is a contrast to IRIS and JOWA in how tokenized predictions are paired with behavior learning. These papers share a broad tokenized-world-model pattern while differing in their training regimes and policy components.

Genie: Generative Interactive Environments (Bruce et al., 2024, Proceedings of the 41st International Conference on Machine Learning) is a forward pointer rather than a peer. Its video tokenizer and autoregressive dynamics are directly relevant to this chapter, but its most distinctive contribution is the inference of latent actions from unlabeled video, the topic of Chapter 52 on Genie and interactive generated environments in Part IX: Foundation and World-Action Models. We mention it here only because it belongs to the same family and because the same questions about action interfaces and closed-loop evaluation apply.

Notice what we have not claimed: we have not claimed that any of these models transfer zero-shot to new games, that any of them solve long-horizon planning, or that they are the best models in their respective benchmarks. We have simply named a few primary sources for the broad design pattern.

The phrase "the model transfers across games" hides several distinct mechanisms. Each one can fail independently.

• Reusable visual codes. A tokenizer trained on many games may reuse visual features such as edges or sprite shapes. Whether this helps on a new game depends on its imagery and on whether those features preserve task-relevant details; measure visual transfer separately.
• Shared dynamics priors. The model may reuse patterns such as continued motion or contact followed by collection. This may help when games share transition rules, but transfer can fail when their rules differ, as between a projectile shooter and a turn-based puzzle.
• Transferable action semantics. Some controls, such as "up", may have aligned effects across games, while a shared action ID need not mean the same thing everywhere. Overlapping controls can transfer; ambiguous or game-specific ones require context, masking, or adaptation. A latent action alphabet or learned action embedding is another possible interface.
• A shared decision head. The policy or value head trained alongside the model can transfer if the semantics of "good" and "bad" are sufficiently similar across games. This is fragile: a decision head that plays Pac-Man well is not obviously a good head for a racing game.

Each of these can succeed while the others fail, so an ablation study should isolate them. If the visual codes help but the dynamics priors do not, you want to know, because the fix is different in the two cases. A transfer result that reports a single aggregate number hides which mechanism produced the gain.

Different transfer mechanisms rely on different kinds of shared structure. The figure below uses invented scores to illustrate hypotheses one might test as game similarity changes. It is not evidence that one mechanism transfers better than another.

The following protocol makes the transfer claim testable.

1. Hold out environments, not just levels. Reserve entire games from pretraining. Holding out only levels tests intra-game generalization, a different question from cross-game transfer.
2. Specify what is adapted at test time. Does the model fine-tune its tokenizer? Its dynamics? A decision head? An action adapter? Each option defines a different question.
3. Specify adaptation data and evaluation interaction separately. A zero-shot transfer claim normally excludes target-game training or parameter updates, but a controller still interacts with the game during evaluation. Few-shot adaptation uses a stated small target-game dataset. In-context adaptation uses target-game context without weight updates. Report both the adaptation budget and the interactions used to score the resulting controller.
4. Compare against a from-scratch baseline. Training a model on the new game alone helps show whether pretraining produced a transfer gain under the same evaluation budget.
5. Include a no-pretraining ablation. Even a pretrained tokenizer can dominate the effect size; without an ablation you cannot tell whether pretraining the dynamics mattered.
6. Distinguish three transfer questions. Transfer within a game (new levels) is different from transfer to a new game with the same action set, which is different from zero-shot control on a new game with a different action set. Solving one does not imply solving the others.

The following is not a rule, but a rule of thumb: if a cross-game transfer result does not specify the action interface, the data budget, and the level of adaptation, it is not interpretable.