Autoregressive Token World Models › Applications and Results

Categorical sequence world models and related interactive generators illustrate three distinct possibilities: sample-efficient control, generative interaction, and shared multimodal processing. Their results should be compared on their actual architectures and evaluations.

IRIS combines a learned visual autoencoder, a GPT-style dynamics predictor, and an actor-critic. It reports strong Atari 100k aggregate results among approaches without lookahead search; search-based comparisons differ. This demonstrates a useful categorical world-model recipe, not a proof that token loss counts cause sample efficiency. Recurrent decoders also supervise many quantities, so such a causal comparison would need matched information, objectives, and compute.

Genie combines a temporally aware ST-transformer VQ-VAE tokenizer with a learned 1,024-entry codebook, a latent action model, and masked next-frame dynamics. It uses iterative MaskGIT decoding within frames, not ordinary raster next-token generation, and evaluates fidelity, controllability, and policy learning rather than experience alone. Action-unlabeled video broadens possible training data, subject to selection, licensing, quality, and domain constraints; earlier observation-only predictive models already existed. Oasis instead uses a spatial autoencoder and latent diffusion to generate successive frames, reporting 20 frames per second on its specified hardware. Neither result validates a single shared discrete next-token mechanism or guarantees long-term object permanence.

Shared sequence processing is one route toward larger multimodal systems. Chameleon demonstrates discrete text/image early fusion; Gato combines continuous image embeddings with scalar tokens for policy outputs. Shared processing still permits modality-specific tokenizers, masks, and heads. It does not imply one scaling curve across modalities or general superiority over specialized components. Part IX develops these comparisons.