Tokenized Game and Environment Models › Limitations & impact

Tokenized game models use compact discrete observation sequences and supervised predictive losses, but their reachable code space can still be large and autoregressive sampling can be costly. Their limitations arise from representation, data support, action semantics, and the way a planner uses their predictions.

The first limitation is the tokenizer. If the current code and available history contain no trace of a decision-relevant variable, the dynamics model cannot condition on it. In our toy, the count model sees only the current lossy grid, so removing the key tile aliases key-held and key-unheld states. Reconstruction-oriented objectives, especially a uniform pixel loss, can underweight small task-relevant details relative to large surfaces. A task-aware loss, longer action history, or a separate symbolic channel may help, but each needs its own test of whether the required distinction survives.

The second limitation is the action interface. A shared token model needs to interpret the action it receives. Even in one game, latency, action repeat, and variable frame rates can create a mismatch; multiple games add differing controls. Cross-game transfer results therefore need to state what is shared and what is adapted at test time. A policy trained inside a token model can earn high imagined returns yet perform poorly in the real game when it exploits an action-interface or transition-model error.

The third limitation is data support. When a model is trained offline on trajectories, its reliable coverage depends on those trajectories. Expert demonstrations may concentrate on successful paths and omit alternative actions; exploratory data may visit different states without reaching a task objective. Neither collection strategy guarantees coverage of the compound action sequences that a planner wants to evaluate. Our toy world exhibits this cleanly: the restricted dataset covers most of the grid but completely misses the pickup transition, so the model has no entry for the state it needs. Real datasets are messier: coverage is partial, action distributions are uneven, and rare events are rare for a reason. You should not assume that a large dataset implies coverage of the specific branch you care about. This is the same concern we discussed in Part VIII, Ch 6: Offline and Conservative Model-Based RL, where the mismatch between the data distribution and the query distribution motivates conservative evaluation.

The fourth limitation is evaluation. Images are easy to inspect, but a visually coherent rollout can hide a wrong reward, a missed termination, or an inconsistent level layout. No single scalar certifies a world model: reconstruction, token likelihood, event prediction, reward, action sensitivity, calibration, long-horizon consistency, and real closed-loop return answer different questions. A result reported on one metric should not be read as a result on the others.

Tokenized game models offer a useful testbed. Existing game trajectories can train predictive models, and discrete codes can reduce sequence length compared with raw pixel tokenization. Resettable environments support repeatable policy-return tests; comparing counterfactual branches from the same interior context additionally needs snapshot/restore, reproducible replay, or equivalent state control. Sampling cost still depends on model size, token count, and decoding scheme, so cheap imagined trajectories are not guaranteed. These systems let us test ideas from Part VII: Planning and Agency and Part VI: Learning World Models at Scale under controlled conditions.

The practical implication is that a tokenized game model should be described with the same care as any other world model. Which data was it trained on, with which behavior policy, over which action interface, and with what testing regime? Under what conditions does it transfer, and under what conditions does it fail? And, most importantly, what evidence do we have that the learned transition law preserves the distinctions an agent needs, rather than merely reproducing the appearance of the world?

This is the setup for the next chapter. Our toy world used a hand-designed tokenizer: the grid of tile types is already discrete, and we only had to decide whether to include the key. Real environments are not so kind. Chapter 50, Pretrained Visual Models and DINO-WM, in Part IX: Foundation and World-Action Models, changes the representation choice fundamentally: it keeps a continuous feature space learned by a pretrained visual model, and it predicts in that space instead of predicting discrete codes. The intuition we built here, that a representation is only as useful as the decisions it supports, carries over directly. The mechanisms do not.