Autoregressive Token World Models

Imagine supplying a game model with an initial agent position shifted by seven cells modulo sixteen. Does the model return to the observed trajectory, follow a different but valid trajectory, or make further mistakes? We will test those possibilities in a small deterministic world. The preceding chapters introduced recurrent state and transformer memory; here we focus on a particular interface for dynamics: discrete observation and action tokens predicted by a causal transformer.

A world model predicts environment-relevant futures from available observations and actions. A recurrent model carries a learned summary; a token model can retain an explicit recent token record and compute predictions from it. Both use conditional sequence factorizations, but lossy tokenization changes the predicted outcome space: a distribution over token grids is not identical to a distribution over raw images. Causal-transformer optimization and attention machinery are reusable, while modality alignment, tokenization, loss masks, and evaluation still need design. Whether scaling behavior transfers is an empirical question.

IRIS demonstrates categorical next-token world modeling for Atari control. Related interactive generators use different decoders: Genie generates discrete video frames with iterative masked-token prediction, while Oasis generates frames autoregressively using latent diffusion. Temporal autoregression, discrete representation, and within-frame next-token decoding are separate choices. We will use IRIS's general categorical sequence pattern for the miniature and distinguish the alternatives when they arise.

The pipeline has three stages, and each gets its own treatment in this chapter:

• Tokenization: observations and actions are mapped to symbols. A fully unified design combines disjoint modality vocabularies; hybrid systems may retain continuous inputs or separate prediction heads.
• Next-token dynamics: a causal transformer learns the conditional distribution over future world tokens given the past token stream, including action tokens that we inject along the way.
• Rollout and use: the model generates hypothetical futures by feeding its own predictions back as inputs, and an agent (in Part VII) queries those futures to make decisions.

The apparent simplicity hides design decisions. A tokenizer can discard distinctions needed by a task, although history or other modalities may retain information absent from one frame. Serialization and masks determine which inputs a prediction may read; chronological ordering alone does not establish causal identification. Under teacher forcing, training uses recorded histories. Imperfect autoregressive models may generate different histories at deployment, and their errors can influence later predictions. We will measure rollout errors and corrective fine-tuning without treating every mismatch or declining exact-match curve as proof of exposure bias.

The runnable miniature keeps the original three-entity environment, two training budgets, self-generated-context fine-tuning, and a one-token intervention. Matched teacher-forced metrics, an additional-training control, and an exact environment predictor help separate what these experiments do and do not establish.

Before diving into tokenizers, it is worth being explicit about what changes relative to the recurrent state-space models of Part V: World-Model Architectures.

A recurrent world model carries a fixed-width state $h_t$. At fixed numerical precision its storage capacity is finite, but the state can still be a sufficient statistic for a task. A strict window-conditioned token model instead recomputes predictions from the most recent $W$ token IDs and their positions. Those IDs are retained exactly, not the pre-tokenization observations. Rolling multilayer KV caches are a different implementation: retained contextual representations can encode information from tokens whose original entries have since been evicted.

Both designs allocate finite resources to an unbounded past. The recurrent summary may preserve selected information from arbitrarily long ago; the explicit token buffer preserves a bounded record. Neither guarantees successful prediction: representation loss, available context, learned attention, and optimization all matter. Direct availability of a token is not the same as retrieving its useful content.

This difference has consequences worth internalizing early:

• Memory quality: a strict token buffer preserves its stored symbols exactly. A recurrent state can preserve selected old facts or an adequate filtering statistic without preserving the full record. A relevant event outside a recomputed buffer is unavailable only if no retained evidence represents it. Tasks needing recent fine detail and tasks needing distant events therefore suggest different tests, not an automatic architecture ranking.
• Per-step compute: at fixed dimensions, a recurrent update has $O(1)$ work in elapsed history. A cached new-token attention query has $O(W)$ mixing work; recomputing dense attention over the entire prefix costs $O(W^2)$ per generation step. KV caching reuses earlier projections because a causal model's earlier states do not depend on future tokens, provided positions and other state conventions remain consistent. The cache uses $O(W)$ storage. Dense full-sequence training has quadratic pairwise attention work, distinct from the linear cached-query cost.

The next cell plots illustrative asymptotic growth, with each curve separately normalized to one at the smallest window. It is not a timing benchmark or a claim that the methods have equal costs at that window. It also computes cache storage for one sequence with full-width keys and values, width 1,024, 12 layers, and two-byte entries; other head layouts, batch sizes, or precisions change that accounting.

• Training signal: next-token prediction provides a loss term at each supervised token position. A $32\times32$ image grid has 1,024 such terms per frame. Recurrent image models can also supervise many pixels, rewards, and latent quantities; naming their sum one loss does not reduce its information content. Token supervision is a convenient objective, not a proof of more useful gradients or superior sample efficiency. Auxiliary objectives may still help retain task-relevant information.
• Discreteness: categorical probabilities define an explicit predictive distribution, but need not be calibrated or separate epistemic from environmental uncertainty. Sampling draws symbols rather than averaging their decoded values. Learned visual quantizers can lose information; tokenization of already discrete variables can be exact. Continuous encoders also have architectural and learned bottlenecks. No downstream predictor can recover an absent distinction from unchanged inputs alone, though other retained evidence may make it inferable.

A token world model is only as good as its tokenizer. This section covers three common tokenization design problems: turning images into tokens, turning actions into tokens, and fusing many modalities into one vocabulary.

Tokenization is not necessarily learned or lossy: our toy uses an exact hand-designed mapping. The visual tokenizers below do compress observations. If two complete retained token histories are identical but require different correct predictions or decisions, no downstream model can always distinguish them. A variable missing from one encoded frame may still be inferable from other frames or actions. Temporal stride sets the latent grid rate, not by itself the finest event timing the codes can preserve or the decoder can reconstruct.

A $256\times256$ RGB image contains 196,608 scalar channel values. iGPT applied autoregression to small images represented by clustered pixel colors, with VQ-VAE compression for larger inputs. At one color token per pixel, a $64\times64$ frame requires 4,096 ordinary sequential decoding steps. Each step evaluates a decoder, although KV caching avoids recomputing the full prefix. Learned visual compression reduces the number of positions by encoding an image or clip as a smaller code grid.

The workhorse is the vector-quantized variational autoencoder (VQ-VAE), introduced by van den Oord, Vinyals, and Kavukcuoglu (2017). It works in three stages:

1. An encoder $E$ maps an image $o$ to a downsampled feature map $z = E(o)$. A $256 \times 256$ frame with 8× downsampling becomes a $32 \times 32$ grid of feature vectors, one per spatial cell.
2. Each feature vector $z_i$ is quantized: we replace it with the nearest entry in a learned codebook $\{e_1, \dots, e_K\}$:

where:

• $z_i$: the feature vector at spatial cell $i$ of the encoder output
• $e_k$: the $k$-th entry of the learned codebook, one of $K$ entries in total
• $\lVert \cdot \rVert_2$: the Euclidean norm, so $k^\star$ is the index of the codebook entry closest to $z_i$
• $\tilde{z}_i$: the quantized replacement for $z_i$, namely the selected codebook entry

The assignment geometry is easiest to see in two dimensions. The next cell simulates a small feature space: five clusters of encoder outputs (the analogue of one spatial cell's feature vector across many images), a codebook of 12 entries, and the nearest-neighbor rule from the equation above. Everything is deterministic given the seed.

Video tokenizers can encode time as well as space. MAGVIT uses 3D operations; a compressed code can summarize multiple frames, for example a $2\times16\times16$ spatiotemporal block. Temporal stride spaces latent positions farther apart, but a code may retain within-block timing and a decoder may output every frame. Two-frame encoding is therefore not equivalent to dropping every other frame. Loss of timing depends on capacity and the learned representation. A noncausal encoder that waits for a second unseen frame adds latency; causal or overlapping schemes can have different online interfaces. Actual downsampling aliases unresolved temporal frequencies unless filtering or prior constraints make them recoverable. These distinctions connect token design to the observation-design questions in Part III.

Continuous-token designs omit codebook rounding but may retain other projection or compression losses. A continuous head can be deterministic, Gaussian, or an iterative generative module. MAR is an autoregressive image model with continuous tokens and a diffusion loss. Iterative sampling adds work, but continuous outputs are not inherently more expensive than categorical decoding across many positions. The relevant comparison includes evaluations per output, width, parallelism, representation quality, and the task.

Actions enter the token stream too. Many environments have naturally discrete actions; many do not.

• Natively discrete actions need an agreed symbol mapping and timing convention, but no continuous-value quantization. Atari's action space has fewer than 20 actions; this makes token mapping straightforward. IRIS evaluates on Atari 100k, but this convenience alone does not explain its historical benchmark choice.
• Continuous actions must be quantized only for a fully categorical interface. RT-1 uses 256 bins per scalar action variable; its arm variables describe position, orientation, and gripper opening rather than seven joint angles. Any seven-scalar command takes seven tokens under one-token-per-variable encoding. Gato bins scalar observations and actions into 1,024 levels, while images use continuous patch embeddings. Hybrid observation-token models can instead accept or emit continuous actions.

Binning creates several distinct costs. Boundary jitter occurs when a varying command crosses a quantizer threshold repeatedly: small smooth changes become switches between adjacent output levels. This can add higher-frequency components to the command. It is not necessarily bang-bang control, and whether the physical trajectory oscillates depends on the plant and controller.

The next cell illustrates command quantization without a learned policy or a physical plant. A sinusoid crosses an interior bin edge in an eight-bin partition of $[-0.5,0.5]$; the figure shows command discontinuities, not a simulated trajectory.

Precision depends on dynamics as well as bin width. With midpoint decoding of an in-range scalar command, quantization error is bounded by half a bin width. That is not a bound on position error or a proof of unreachable targets: target alignment, feedback, integration, and execution horizon matter. Latency arises when action variables are decoded sequentially. Seven categorical variables require seven ordinary decoding steps in that design, but parallel heads or externally supplied candidate actions need not. Candidate rollouts also cost computation, although batching can share hardware throughput; Oasis's video-generation speed is not evidence about seven-token action decoding.

Continuous action heads can share an observation-token backbone. Coarse-to-fine tokens add precision with additional codes, but task-adaptive allocation requires an extra mechanism. Genie's latent action model instead uses prior frames and the next frame to learn compact transition codes without labeled actions. Such a code can provide a useful controllable interface, as evaluated in Genie, but observational frame changes can also reflect camera motion or exogenous events. Compactness alone does not identify a causal intervention. We return to this distinction in Part VI.

A fully unified categorical design can represent observations, actions, rewards, and termination flags in one stream. Decision Transformer instead conditions a policy on embedded return-to-go scalars; this is not the same as predicting categorical reward symbols. An integrated reward predictor can support planning if it predicts the task's relevant reward accurately. Termination is supervised only when included as a target, and an integrated head does not remove the need to evaluate planning decisions.

A fully discrete implementation can use disjoint token ranges and optional modality-boundary markers. Chameleon demonstrates early-fusion text and image modeling. Gato uses a shared policy transformer over a mix of discrete symbols and continuous image-patch embeddings, without predicting image observations. Modality-specific processing can also be fused later at an intermediate or output representation. These designs share some sequence machinery without being identical categorical world models.

Why go through the trouble? Three reasons:

• One objective, shared processing. Supervised discrete output positions can share cross-entropy. Tokenizers may require reconstruction or other objectives, and input-only modalities need no output loss. Adding a sensor can also require synchronization, missing-data conventions, masks, positional encodings, loss balancing, and evaluation. A common optimization loop is reusable, but equal loss forms do not imply equal scaling curves.
• One context. Shared attention provides a path for cross-modal conditioning without a dedicated pairwise fusion module. A past command can affect a frame prediction when it remains accessible under the context and mask. Learning that dependency still requires suitable data and optimization. Controlled tasks and modality ablations help distinguish genuine conditioning from shortcuts.
• Reusable infrastructure. Causal-transformer optimization, data parallelism, and cache techniques are reusable. Context-parallel kernels must support the chosen masks; quantization needs calibration and accuracy checks; speculative decoding needs compatible draft probabilities and action-conditioning semantics. Reuse is a practical advantage, not a promise that every component transfers unchanged.

Modality token counts can differ substantially: a 1,024-token image may sit beside one scalar token. Equal logits would give the image keys 1,024 times the scalar's total attention mass, but learned logits can overcome this count difference. Comparable information density is not required. Modality embeddings identify type and positions identify order; whether either helps an imbalance-related learning problem needs measurement. Token budgets, masks, and training mixtures are design choices rather than an intrinsic one-dimensional bottleneck.

We now formalize what the model learns. Let a rollout be serialized as a single token stream interleaving observations and actions:

where:

• $x$: the token stream for one rollout
• $o_t$: the observation tokens emitted at step $t$
• $a_t$: the action tokens at step $t$, ordered before observation block $o_{t+1}$

Each $o_t$ and $a_t$ may itself expand into several tokens. A causal transformer with parameters $\theta$ defines an autoregressive factorization over any such stream:

where:

• $x_1, \dots, x_n$: the tokens of the stream $x$, listed in sequence order
• $x_{<t}$: all tokens preceding position $t$, that is, $(x_1, \dots, x_{t-1})$
• $p_\theta(x_t \mid x_{<t})$: the categorical distribution over the vocabulary that the transformer, with parameters $\theta$, assigns to the next token
• $n$: the total number of tokens in the stream

The chain rule permits any ordering of a joint distribution. A causal next-token implementation additionally needs shifted targets: the input at the prediction position must not already contain the target. The usual mask allows self-attention and only excludes later input positions. Finite context restricts the represented conditionals, whereas the unrestricted chain rule itself makes no independence assumption. Expected negative log-likelihood is minimized by the true conditional distribution when it is representable and optimized in expectation; the optimum can be sharp or diffuse.

For prediction under externally specified actions, define the causally conditioned rollout law below. This is not generally the observational conditional obtained by selecting trajectories with an entire future action sequence: under an adaptive logging policy, future actions can reveal earlier observations. The causal rollout construction instead supplies actions without that retrospective selection. Calling its factors true intervention dynamics additionally requires adequate conditioning information, action support, and no relevant hidden confounding.

where:

• $o_{1:H}$: the predicted observation blocks at steps 1 through $H$, with $o_t=(o_t^{(1)},\ldots,o_t^{(m)})$
• $\Vert$: causal conditioning on supplied actions, not selection on future logged actions
• $H$: the rollout horizon
• $o_0$: the initial observation tokens, given as context rather than predicted
• $a_{0:H-1}$: the externally specified actions for steps 0 through $H-1$
• $m$: the number of tokens per observation block
• $o_t^{(j)}$: the $j$-th token of the observation block at step $t$
• $o_{<t}$: all observation tokens from steps before $t$
• $a_{<t}$: actions before $t$, including $a_{t-1}$; future actions are excluded by the causal rollout construction, not by an identity for arbitrary observational conditioning
• $o_t^{(<j)}$: the first $j-1$ tokens of observation block $t$ itself

Read this carefully, because it encodes several design decisions:

• Actions are conditions in this objective. We mask action targets and learn observation-token factors. Joint models can also learn behavior-policy probabilities without mathematically invalidating the same observation conditionals. Either objective needs support and causal assumptions before its predictions are trusted under intervention. In our toy, controls are randomized independently of initial states and the transition mechanism is known; a planner in another dataset cannot assume that masking actions removes confounding or provides evidence for unseen actions.
• Within-frame dependence is a decoding choice. In raster next-token decoding, a token can read earlier tokens in its frame and accessible earlier frames. Independent simultaneous outputs cannot represent residual correlations unless those are resolved by conditioning or other latent structure. Iterative masked decoding permits a different within-frame information graph. Neither access pattern guarantees coherent generated frames.
• Available context determines usable evidence. For strict recomputation, state includes the last $W$ token IDs and positional metadata. Rolling caches can carry additional contextual information. A short window suffices only when the tokenized observation-action record preserves a sufficient predictive state. In partial observability, a Bayesian posterior would summarize history under a specified model; learned transformer activations need not equal that posterior or be calibrated. Windowed prediction can approximate belief-based prediction using accessible evidence without explicitly representing the full belief.

Training is by maximum likelihood under teacher forcing:

Under teacher forcing, the model is trained to maximize the likelihood of the true world tokens given their true contexts:

where:

• $\mathcal{L}(\theta)$: the expected negative log-likelihood of the world tokens under parameters $\theta$, to be minimized
• $x$: a token stream drawn from the training data, so the expectation is taken under the data distribution rather than the model's own
• $x_t$: the target token at position $t$, with $x_{<t}$ the true context fed to the model as input
• the sum ranges only over positions whose target is a world (observation) token; action tokens are masked out of the loss

Teacher forcing gives one loss term per supervised world token, uses recorded rather than self-generated histories, and permits layerwise-parallel evaluation across sequence positions. It avoids error feedback in the training inputs, but does not guarantee numerical stability or sufficient representations. Dense attention still has quadratic pairwise work. When learned conditionals reproduce the data process under matched actions, generated and recorded histories can have the same distribution; mismatch is a risk of imperfect models and changed deployment conditions, not an unavoidable consequence of teacher forcing.

It is worth pausing on a distinction this book keeps returning to, because token world models blur it more than most. The model above predicts observations: given a history and a candidate action sequence, it rolls out a distribution over future observation tokens. That is an open-loop forecasting capability, and everything in this chapter is about its quality.

Using predictions is a separate task from producing them. Model-predictive control searches candidate actions, executes an initial action, then replans from new evidence. Fresh observations can correct visible drift, but robustness still depends on model accuracy, sensing, constraints, and controller design. Differentiable planning through discrete choices needs an expectation, relaxation, or gradient estimator; an agent can also learn from imagined experience or distill a planner's actions. Search need not find a global optimum and may exploit overestimated rewards. Conversely, a model can retain enough task-relevant structure despite poor texture reconstruction. Predictive fidelity and decision usefulness therefore require separate tests, without assuming a general correlation.

A video has spatial and temporal axes, while a token transformer consumes a linear ordering. The ordering and mask determine available inputs; chronology is particularly convenient for online generation. Any order can factor an offline clip joint in principle. For a finite model, ordering affects sequence distances and inductive bias, but accuracy differences require evidence rather than an appeal to physical causality.

For a clip of $F$ frames with $P\times P$ tokens per frame, consider two instructive orders.

Frame-major (raster) order concatenates complete frames: all $P^2$ tokens of frame 0 in raster order (left-to-right, top-to-bottom), then all tokens of frame 1, and so on. Consequences:

• Frame-major order places all preceding-frame tokens earlier in the sequence. They are eligible context when retained and allowed by the attention mask. Within-frame raster order remains a modeling choice rather than a physical law.
• Within a frame, earlier raster positions include all previous rows and earlier columns of the current row. Prior-frame information can help complete the current frame, but access alone does not establish that motion coherence is learned before appearance coherence or that one task is easier.
• Actions and rewards can be inserted at frame boundaries. This exposes preceding actions to the next-frame predictor; causal interpretation and control usefulness still require correct timing, masks, support, and dynamics.

Position-major order puts all times for one spatial position before the next position. It is a valid offline clip factorization. It is not automatically a valid online forecasting protocol: supplying observed future frames would leak information unavailable at prediction time. Same-frame positions are separated by the clip length, and simple once-per-frame action insertion is inconvenient. Actions can still be supplied through a separate interface or repeated conditioning. The ordering changes sequence distances and decoding schedule, not the existence of temporal structure.

The figure below makes the difference concrete on a toy clip: three frames, each flattened to four tokens. Numbers show each token's position in the sequence.

Beyond the flattening order, several attention structure variants are used to manage cost and inductive bias. The unifying theme is that full attention over long token streams is quadratic, and every variant below is a different guess about which attention edges can be sacrificed without losing the interactions that matter:

• Full causal attention has quadratic pairwise work. Twenty seconds at 30 frames per second and 32 tokens per frame gives 19,200 tokens. In an illustrative dense block with width $d$ and a $4d$ MLP, attention mixing takes about $2S^2d$ multiply-accumulates versus $12Sd^2$ for projections and the MLP, giving a crossover near $S=6d$. Triangular evaluation, other expansion ratios, and kernels change that estimate. Cache storage also depends on width, head layout, depth, batch, and precision.
• Sliding-window attention removes direct attention edges beyond a per-layer span. Stacked layers and contextual cached states may transmit older information indirectly, so a local mask is not equivalent to deleting all history outside that span. Test whether a needed variable survives in any accessible representation rather than treating its original token's eviction as proof of forgetting.
• Factorized spatiotemporal attention alternates within-frame spatial communication with causal same-location temporal communication. For $F$ frames and $N$ spatial tokens, dense mixing work is $O(FN^2+NF^2)$ at fixed width, rather than $O(F^2N^2)$ for full attention. Composed spatial and temporal paths can convey cross-location motion, but do not guarantee coherence. Genie's ST-transformer uses this structure with iterative MaskGIT next-frame decoding; its bidirectional spatial layer is not the raster within-frame factorization above.
• Temporal token context lets independently encoded frames contribute cross-frame evidence. IRIS uses 16 visual tokens per $64\times64$ frame and a 20-step context. A single instantaneous frame need not identify velocity, although blur or other cues may help; previous frames supply displacement information. DQN similarly stacked four preprocessed frames. Additional historical frames increase stored context tokens, not newly emitted tokens per step. Relative cost versus a temporal encoder depends on both implementations.

Re-grounding supplies new real observations during deployment; it can correct visible prediction errors but not necessarily hidden-state errors or consequences of past bad actions. Pure generative environments lack such a reference stream. GameNGen is a separate example of robustness training: it conditions a Stable Diffusion-based generator on prior actions and frames and adds Gaussian noise to encoded context frames during training. This is not deployment-time re-grounding. Short-horizon use may need less rollout accuracy, but even one-step errors can make a controller choose harmful actions.

Pure teacher forcing uses recorded histories; autonomous rollout uses generated world tokens after its supplied initial context. The question is whether those histories differ in distribution and how the model behaves on them.

During rollout, world-token contexts are generated conditionally on the initial observation and supplied actions; those supplied inputs are not themselves sampled from the model. Model errors can shift later contexts and provoke further errors, but may also leave them in distribution or self-correct. Flawless-path probability is a different quantity. Let $S_t$ denote success at step $t$ and $p_t=\Pr(S_t\mid S_1,\ldots,S_{t-1})$. Then the chain rule gives

where:

• $p_t$: the probability of exact success at step $t$ conditional on all earlier successes
• $H$: the rollout horizon
• $p$: a constant conditional success probability along surviving prefixes; independent equal-accuracy steps are one sufficient special case

At $p = 0.99$ and $H = 100$, that is about 37%. At $p = 0.95$, under 1%.

The next cell plots the illustrative constant-success law $p^H$ across 400 steps. Later we estimate empirical flawless survival directly; we do not assume the measured errors are independent.

Recovery can improve current-step accuracy but cannot restore a rollout's flawless-so-far status after a failure. Dependence can raise or lower survival relative to a product of marginal accuracies. For example, perfectly shared success events give survival $p$ at every horizon, whereas independent successes give $p^H$. Pairwise correlation alone does not determine higher-order intersections. A uniform conditional failure probability at least $\delta>0$ on every surviving prefix implies survival at most $(1-\delta)^H$; merely nonzero marginal errors do not establish that bound.

Visual artifacts require diagnosis. Biased transitions can drift, depending on propagation gains; inconsistent patch predictions can impair coherence. Blur may arise from tokenizer reconstruction or an averaging decoder, but categorical sampling selects token IDs rather than averaging their decoded values. A token-only rollout feeds those IDs back, not necessarily a blurred decoded image. An object may disappear through representation or prediction errors even while its evidence remains in context; eviction matters only when no retained information preserves the needed distinction.

It is important to separate two error sources that produce similar-looking drift, because they have different cures:

• Model error and feedback: clean one-step tests can reveal prediction errors, while rollout tests additionally expose their composition under feedback. Properly labeled generated-context training may help, but improvement and its cause require controls.
• Environmental uncertainty: a correct stochastic model reproduces a conditional distribution, not the particular realization in a recorded trajectory. Independent valid draws can disagree even with no distribution mismatch. Compare generated samples with repeated environment outcomes or a validated conditional reference; dispersion of model samples alone cannot distinguish correct uncertainty from a diverse wrong model.

The mitigation toolbox connects sequence modeling and control. Each intervention has its own target, information requirements, and possible side effects; no general ranking follows from the list.

• Scheduled sampling (Bengio et al., 2015) replaces some recorded prior inputs with model outputs during training. It helped particular reported tasks, but changes ordinary data likelihood. Huszár's analysis shows that its perturbed objective can be statistically inconsistent. Treat its schedule and outcomes as empirical choices rather than a universally principled repair.
• DAgger and generated-context training. DAgger aggregates expert-labeled states visited by a learner and reduces imitation learning to no-regret online learning. Its guarantees concern a suitable policy iterate under that reduction's assumptions, not monotone improvement of arbitrary neural rollouts. It needs an appropriate labeling oracle, not necessarily a simulator. Our later procedure is only inspired by this idea: it performs online corrective fine-tuning without dataset aggregation and uses reference-trajectory targets, so the DAgger theorem does not transfer.
• Input perturbations and output regularization target different mechanisms. Context perturbation trains on altered inputs; label smoothing regularizes output probabilities. Numerical token-ID distance has no inherent semantic meaning. Robustness to perturbations in a defined observation or representation metric need not extend to large changes or unstable multi-step propagation.
• Multi-step losses supervise predictions farther into a rollout. Continuous intermediates can support pathwise gradients; hard categorical choices require a specified estimator, relaxation, or surrogate. Shorter horizons may ease gradient estimation, while longer horizons cost more and can expose different errors. Neither horizon guarantees useful gradients or a valid likelihood interpretation; the exact objective matters.
• Decoding choices such as temperature, top-$k$, or nucleus truncation change diversity and mode preference. Low model probability does not mean an event is wrong, and a confident argmax can be wrong. Conservative decoding may suppress errors or valid rare outcomes; repetition can occur in some regimes. Measure these effects on the intended distribution rather than assuming coherence improves.
• Fresh observations can reduce purely open-loop drift when the environment remains available. Re-reading generated history can preserve earlier information but supplies no independent truth correction. Replanning with observations does not eliminate model bias or confer robustness without sensing, estimation, and control assumptions.

Evaluate interventions against the properties they target. Paired image distances, distributional video metrics, exact-match rates, and controller returns answer different questions. A single prespecified metric can be useful evidence about one property, but is not a complete assessment of a world model.

We now build a deterministic one-dimensional world and compare prediction under recorded and generated contexts. It has three entities:

• An agent at a fine position $ap \in \{0, \dots, 15\}$ that moves left, right, or holds, and is clipped at the walls.
• Ball 1, which moves one cell to the right each step, wrapping around.
• Ball 2 moves three cells per step with wraparound. Both balls' discrete position sequences repeat after 16 steps because 1 and 3 are coprime to 16; ball 2 has three times the displacement per step, not a shorter sampled repetition period.

The miniature uses the interleaved-token pattern of some categorical world models, without a learned visual tokenizer. An initial observation has three tokens; each transition adds an action followed by agent and ball positions. The disjoint ranges serve as exact encodings of these variables.

Action-before-observation order makes the control available at every next-frame prediction, and action targets will be masked from the loss. The two independent balls demonstrate multiple temporal processes sharing a stream; they do not simulate occlusion, visual rendering, or interacting-object physics.

We next compare self-generated-context corrective fine-tuning with continued teacher forcing. Both start from the same 200-step weights and use the same newly sampled states and action sequences, 60 optimizer updates, batch size, learning rate, and 16-step supervision horizon. Both evaluate every prefix separately, so their forward/backward call counts and token lengths are matched. Wall-clock timing can still differ. Evaluation extends to 24 steps, beyond the fine-tuning horizon.

The generated-context branch predicts along its own token history but receives targets from the original reference trajectory. These labels ask it to return to that trajectory, not to obey the environment transition starting from each hallucinated state. The original initial state and known actions make reference labels available; fixed recordings could supply the same labels without an interactive oracle. Some generated contexts can be ambiguous or inconsistent, so useful correction is not guaranteed. No dataset is aggregated, and no DAgger theorem is claimed.

In the seeded CPU run (PyTorch 2.13.0, NumPy 2.4.2), mean frame match over steps 1–16 was 30.3% for the base model, 71.9% after generated-context correction, and 97.6% after continued teacher forcing. Over steps 17–24, the same figures were 7.0%, 1.4%, and 0.05%. All three had zero flawless survival through step 24. The extra teacher-forced updates explain a stronger in-horizon gain than corrective contexts in this run; neither continuation improves final-step accuracy or 24-step flawless survival.

The comparison is intentionally allowed to fail. Additional teacher forcing can improve ordinary fitting, while corrective training may help within its supervision horizon and harm later predictions. Use the printed results for each branch rather than attributing every gain to generated-context training. A rule-based predictor can exactly simulate this finite deterministic system, but that does not prove this neural architecture will learn it perfectly. In stochastic worlds, imperfect exact-path matching can persist even for the true conditional distribution and is not itself exposure bias.

Finally, shift the supplied initial agent position by seven modulo sixteen while keeping the reference trajectory and actions unchanged. Because a modulo shift permutes a uniform initial distribution, the shown starting states remain in distribution. They are wrong relative to the paired reference, not intrinsically invalid inputs.

The oracle control follows the exact dynamics from the shifted initial state. If even this exact predictor disagrees with the original reference, that discrepancy is a consequence of changed evidence rather than exposure bias. A second learned-model score against the shifted oracle trajectory distinguishes fidelity to the supplied alternative state from agreement with the original trajectory. The intervention does not remove ordinary neural prediction errors.

Clean and shifted runs use paired actions and initial states. We measure agent and ball accuracy and count re-synchronization explicitly. Generated and observed tokens share the input interface without provenance labels, but this does not force equal reliance on every token. The test concerns sensitivity to initial evidence, a phenomenon that also occurs in recurrent predictors.

In the same 256 paired episodes, the exact shifted oracle re-synchronized with the original agent trajectory at least once in 5 episodes. The learned agent averaged 0.6% accuracy against the original trajectory after the shift, but 91.5% against the alternative trajectory implied by its supplied initial state. Clean and shifted ball-token means were 45.8% and 45.9%. Those near-equal ball means do not establish attention independence.

Agent clipping under common actions can merge distinct positions; wraparound motion preserves circular displacement instead. The exact shifted-state oracle makes this distinction visible without attributing recovery to learned attention. Clipping itself is nonexpansive in Euclidean position and strictly contracts only when it shortens a difference; a larger surrounding transition can still expand differences. Learned-model accuracy changes do not establish whether attention routes information between independent variables. In video, spatial and temporal paths may spread errors, but raster order does not imply neighboring patches are affected first. These are hypotheses for intervention tests, not diagnoses from appearance alone.

Categorical sequence world models and related interactive generators illustrate three distinct possibilities: sample-efficient control, generative interaction, and shared multimodal processing. Their results should be compared on their actual architectures and evaluations.

IRIS combines a learned visual autoencoder, a GPT-style dynamics predictor, and an actor-critic. It reports strong Atari 100k aggregate results among approaches without lookahead search; search-based comparisons differ. This demonstrates a useful categorical world-model recipe, not a proof that token loss counts cause sample efficiency. Recurrent decoders also supervise many quantities, so such a causal comparison would need matched information, objectives, and compute.

Genie combines a temporally aware ST-transformer VQ-VAE tokenizer with a learned 1,024-entry codebook, a latent action model, and masked next-frame dynamics. It uses iterative MaskGIT decoding within frames, not ordinary raster next-token generation, and evaluates fidelity, controllability, and policy learning rather than experience alone. Action-unlabeled video broadens possible training data, subject to selection, licensing, quality, and domain constraints; earlier observation-only predictive models already existed. Oasis instead uses a spatial autoencoder and latent diffusion to generate successive frames, reporting 20 frames per second on its specified hardware. Neither result validates a single shared discrete next-token mechanism or guarantees long-term object permanence.

Shared sequence processing is one route toward larger multimodal systems. Chameleon demonstrates discrete text/image early fusion; Gato combines continuous image embeddings with scalar tokens for policy outputs. Shared processing still permits modality-specific tokenizers, masks, and heads. It does not imply one scaling curve across modalities or general superiority over specialized components. Part IX develops these comparisons.

The limitations are the mirror image of the strengths, and they deserve plain speaking.

Training and rollout contexts can differ. Teacher forcing avoids generated inputs during training but does not guarantee stable optimization. Rollout interventions trade data, compute, statistical bias, and gradient quality; they can help or harm. Our controlled fine-tuning comparison reports both branches rather than assuming improvement. Fresh observations can reduce visible drift when available, while pure generative environments must be evaluated without that correction stream.

Finite token budgets constrain representation. For a frame encoded as $N$ tokens from a $K$-symbol vocabulary, $I(O_t;Z_t)\le N\log_2K$. A $W$-token record has entropy at most $W\log_2K$ under the same vocabulary, before adding any other state or continuous inputs. Neither is a one-frame bound on everything the model can learn about the world. More residual stages lengthen the sequence; larger codebooks raise per-symbol capacity but also affect embedding and output-head costs. Spatial and temporal strides set token grids, not necessarily decoded resolution. Test retention of task-relevant distinctions rather than relying only on generic reconstruction quality.

Memory has implementation-dependent costs. A recomputed finite buffer stores token IDs exactly; rolling caches can carry older contextual information. A needed variable is absent only if no accessible representation preserves it. At fixed dimensions cached attention work per new token is linear in retained context, while full-prefix recomputation is quadratic. Longer context can help or hinder predictions and increases storage and computation. A fixed-width recurrent state offers different resource use and can be sufficient for some tasks.

Action interfaces trade precision and latency. Many robotic systems use continuous multivariate controls. A fully categorical design quantizes them, while continuous or parallel heads remain possible. Command jitter and action-bin error are not themselves bounds on physical trajectory accuracy. Sequential action decoding costs additional evaluations; batching, parallel heads, and external action proposals change the accounting. The interface should be tested against controller requirements, not chosen solely for objective uniformity.

Metrics have limited scope. Framewise LPIPS compares paired images and needs an explicit temporal aggregation for videos. FVD compares distributional feature moments, so correct population distributions match those moments; finite samples or probability calibration alone do not guarantee zero distance. Pooled metrics can also miss whether predictions use the supplied context. Exact-path comparisons can penalize valid stochastic alternatives. None guarantees controller performance or excludes model exploitation. Part XI develops more complete evaluations.

• A fully categorical world model can serialize observations, actions, and optional rewards, then predict world tokens with a causal next-token objective. Tokenization, target shifting, masks, and action semantics still require explicit design.
• Visual tokenizers impose finite representation budgets. VQ-VAE-style, residual, scalar, and lookup-free quantizers differ in geometry and training; no fixed choice guarantees task-relevant sufficiency or full code utilization.
• Actions can use discrete symbols, binned scalars, continuous heads, or latent interfaces. Latent transition codes require controllability evidence; action quantization error and sequential decoding cost have task-dependent control consequences.
• Frame-major layouts support simple online action interleaving. Position-major layouts remain valid offline factorizations but need care for online conditioning. Genie's masked next-frame decoder and Oasis's latent diffusion are distinct from raster categorical next-token decoding.
• An explicit token buffer retains a bounded record, while recurrent states and contextual caches can carry selected older information. Exact stored IDs do not guarantee useful retrieval, and per-step costs depend on the cache and attention implementation.
• Exposure bias concerns a history-distribution mismatch, not every rollout error. Constant conditional success probabilities yield $p^H$ flawless survival, but recovery can only improve current-step match, not restore survival. The miniature compares matched conditioning regimes, two training budgets, corrective fine-tuning against continued teacher forcing, and changed initial evidence against an exact oracle. It does not establish a universal repair or a controller benefit.

The next chapter studies diffusion denoising and flow-based transport as alternatives to categorical prediction. Continuous variants avoid discrete codebook rounding but can retain other compression and numerical losses. Both categorical and continuous models can represent multimodal futures. Total generation cost depends on the number and size of evaluations and available parallelism, not simply whether the output is discrete or continuous.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about autoregressive token world models, tokenization, and rollout behavior.