Autoregressive Token World Models › From Latent States to Token Sequences

Before diving into tokenizers, it is worth being explicit about what changes relative to the recurrent state-space models of Part V: World-Model Architectures.

A recurrent world model carries a fixed-width state $h_t$. At fixed numerical precision its storage capacity is finite, but the state can still be a sufficient statistic for a task. A strict window-conditioned token model instead recomputes predictions from the most recent $W$ token IDs and their positions. Those IDs are retained exactly, not the pre-tokenization observations. Rolling multilayer KV caches are a different implementation: retained contextual representations can encode information from tokens whose original entries have since been evicted.

Both designs allocate finite resources to an unbounded past. The recurrent summary may preserve selected information from arbitrarily long ago; the explicit token buffer preserves a bounded record. Neither guarantees successful prediction: representation loss, available context, learned attention, and optimization all matter. Direct availability of a token is not the same as retrieving its useful content.

This difference has consequences worth internalizing early:

• Memory quality: a strict token buffer preserves its stored symbols exactly. A recurrent state can preserve selected old facts or an adequate filtering statistic without preserving the full record. A relevant event outside a recomputed buffer is unavailable only if no retained evidence represents it. Tasks needing recent fine detail and tasks needing distant events therefore suggest different tests, not an automatic architecture ranking.
• Per-step compute: at fixed dimensions, a recurrent update has $O(1)$ work in elapsed history. A cached new-token attention query has $O(W)$ mixing work; recomputing dense attention over the entire prefix costs $O(W^2)$ per generation step. KV caching reuses earlier projections because a causal model's earlier states do not depend on future tokens, provided positions and other state conventions remain consistent. The cache uses $O(W)$ storage. Dense full-sequence training has quadratic pairwise attention work, distinct from the linear cached-query cost.

The next cell plots illustrative asymptotic growth, with each curve separately normalized to one at the smallest window. It is not a timing benchmark or a claim that the methods have equal costs at that window. It also computes cache storage for one sequence with full-width keys and values, width 1,024, 12 layers, and two-byte entries; other head layouts, batch sizes, or precisions change that accounting.

• Training signal: next-token prediction provides a loss term at each supervised token position. A $32\times32$ image grid has 1,024 such terms per frame. Recurrent image models can also supervise many pixels, rewards, and latent quantities; naming their sum one loss does not reduce its information content. Token supervision is a convenient objective, not a proof of more useful gradients or superior sample efficiency. Auxiliary objectives may still help retain task-relevant information.
• Discreteness: categorical probabilities define an explicit predictive distribution, but need not be calibrated or separate epistemic from environmental uncertainty. Sampling draws symbols rather than averaging their decoded values. Learned visual quantizers can lose information; tokenization of already discrete variables can be exact. Continuous encoders also have architectural and learned bottlenecks. No downstream predictor can recover an absent distinction from unchanged inputs alone, though other retained evidence may make it inferable.