Autoregressive Token World Models › Spatiotemporal Token Layouts

A video has spatial and temporal axes, while a token transformer consumes a linear ordering. The ordering and mask determine available inputs; chronology is particularly convenient for online generation. Any order can factor an offline clip joint in principle. For a finite model, ordering affects sequence distances and inductive bias, but accuracy differences require evidence rather than an appeal to physical causality.

For a clip of $F$ frames with $P\times P$ tokens per frame, consider two instructive orders.

Frame-major (raster) order concatenates complete frames: all $P^2$ tokens of frame 0 in raster order (left-to-right, top-to-bottom), then all tokens of frame 1, and so on. Consequences:

• Frame-major order places all preceding-frame tokens earlier in the sequence. They are eligible context when retained and allowed by the attention mask. Within-frame raster order remains a modeling choice rather than a physical law.
• Within a frame, earlier raster positions include all previous rows and earlier columns of the current row. Prior-frame information can help complete the current frame, but access alone does not establish that motion coherence is learned before appearance coherence or that one task is easier.
• Actions and rewards can be inserted at frame boundaries. This exposes preceding actions to the next-frame predictor; causal interpretation and control usefulness still require correct timing, masks, support, and dynamics.

Position-major order puts all times for one spatial position before the next position. It is a valid offline clip factorization. It is not automatically a valid online forecasting protocol: supplying observed future frames would leak information unavailable at prediction time. Same-frame positions are separated by the clip length, and simple once-per-frame action insertion is inconvenient. Actions can still be supplied through a separate interface or repeated conditioning. The ordering changes sequence distances and decoding schedule, not the existence of temporal structure.

The figure below makes the difference concrete on a toy clip: three frames, each flattened to four tokens. Numbers show each token's position in the sequence.

Beyond the flattening order, several attention structure variants are used to manage cost and inductive bias. The unifying theme is that full attention over long token streams is quadratic, and every variant below is a different guess about which attention edges can be sacrificed without losing the interactions that matter:

• Full causal attention has quadratic pairwise work. Twenty seconds at 30 frames per second and 32 tokens per frame gives 19,200 tokens. In an illustrative dense block with width $d$ and a $4d$ MLP, attention mixing takes about $2S^2d$ multiply-accumulates versus $12Sd^2$ for projections and the MLP, giving a crossover near $S=6d$. Triangular evaluation, other expansion ratios, and kernels change that estimate. Cache storage also depends on width, head layout, depth, batch, and precision.
• Sliding-window attention removes direct attention edges beyond a per-layer span. Stacked layers and contextual cached states may transmit older information indirectly, so a local mask is not equivalent to deleting all history outside that span. Test whether a needed variable survives in any accessible representation rather than treating its original token's eviction as proof of forgetting.
• Factorized spatiotemporal attention alternates within-frame spatial communication with causal same-location temporal communication. For $F$ frames and $N$ spatial tokens, dense mixing work is $O(FN^2+NF^2)$ at fixed width, rather than $O(F^2N^2)$ for full attention. Composed spatial and temporal paths can convey cross-location motion, but do not guarantee coherence. Genie's ST-transformer uses this structure with iterative MaskGIT next-frame decoding; its bidirectional spatial layer is not the raster within-frame factorization above.
• Temporal token context lets independently encoded frames contribute cross-frame evidence. IRIS uses 16 visual tokens per $64\times64$ frame and a 20-step context. A single instantaneous frame need not identify velocity, although blur or other cues may help; previous frames supply displacement information. DQN similarly stacked four preprocessed frames. Additional historical frames increase stored context tokens, not newly emitted tokens per step. Relative cost versus a temporal encoder depends on both implementations.

Re-grounding supplies new real observations during deployment; it can correct visible prediction errors but not necessarily hidden-state errors or consequences of past bad actions. Pure generative environments lack such a reference stream. GameNGen is a separate example of robustness training: it conditions a Stable Diffusion-based generator on prior actions and frames and adds Gaussian noise to encoded context frames during training. This is not deployment-time re-grounding. Short-horizon use may need less rollout accuracy, but even one-step errors can make a controller choose harmful actions.