Autoregressive Token World Models › Limitations

The limitations are the mirror image of the strengths, and they deserve plain speaking.

Training and rollout contexts can differ. Teacher forcing avoids generated inputs during training but does not guarantee stable optimization. Rollout interventions trade data, compute, statistical bias, and gradient quality; they can help or harm. Our controlled fine-tuning comparison reports both branches rather than assuming improvement. Fresh observations can reduce visible drift when available, while pure generative environments must be evaluated without that correction stream.

Finite token budgets constrain representation. For a frame encoded as $N$ tokens from a $K$-symbol vocabulary, $I(O_t;Z_t)\le N\log_2K$. A $W$-token record has entropy at most $W\log_2K$ under the same vocabulary, before adding any other state or continuous inputs. Neither is a one-frame bound on everything the model can learn about the world. More residual stages lengthen the sequence; larger codebooks raise per-symbol capacity but also affect embedding and output-head costs. Spatial and temporal strides set token grids, not necessarily decoded resolution. Test retention of task-relevant distinctions rather than relying only on generic reconstruction quality.

Memory has implementation-dependent costs. A recomputed finite buffer stores token IDs exactly; rolling caches can carry older contextual information. A needed variable is absent only if no accessible representation preserves it. At fixed dimensions cached attention work per new token is linear in retained context, while full-prefix recomputation is quadratic. Longer context can help or hinder predictions and increases storage and computation. A fixed-width recurrent state offers different resource use and can be sufficient for some tasks.

Action interfaces trade precision and latency. Many robotic systems use continuous multivariate controls. A fully categorical design quantizes them, while continuous or parallel heads remain possible. Command jitter and action-bin error are not themselves bounds on physical trajectory accuracy. Sequential action decoding costs additional evaluations; batching, parallel heads, and external action proposals change the accounting. The interface should be tested against controller requirements, not chosen solely for objective uniformity.

Metrics have limited scope. Framewise LPIPS compares paired images and needs an explicit temporal aggregation for videos. FVD compares distributional feature moments, so correct population distributions match those moments; finite samples or probability calibration alone do not guarantee zero distance. Pooled metrics can also miss whether predictions use the supplied context. Exact-path comparisons can penalize valid stochastic alternatives. None guarantees controller performance or excludes model exploitation. Part XI develops more complete evaluations.