Taxonomies of World Models › The State Axis: Choosing Where Prediction Lives

A pixel-space (more generally, observation-space) model learns $p(o_{t+1} \mid o_{1:t}, a_{1:t})$ directly over raw sensory streams: images, audio, proprioceptive readings. Video prediction models are the canonical case. The appeal is obvious: no representation decisions are made for the learner, so nothing task-relevant can be thrown away by a premature bottleneck. Whatever the downstream task needs, if it is visible in the data, the model can in principle carry it. This guarantee is worth taking seriously in domains where nobody knows in advance what will matter: a general-purpose robot operating in unstructured homes cannot enumerate the features its future tasks will require, and pixel-level prediction is the only representation choice that makes no bet on the answer.

The cost is that dynamics in pixel space are violently nonlinear and mostly invariant. When an object moves one pixel to the right, almost every pixel value stays identical, a few change dramatically, and the meaning of the change depends on scene content that the pixel grid does not encode. A function class must be extraordinarily flexible to represent "the cup moved" as a pixel-to-pixel map, and even then it represents it as an incidental correlation rather than a structured fact. Pixel-space models therefore tend to need enormous data and capacity, and their rollouts blur: averaging over uncertain futures in pixel space produces the smeared frames familiar from early video prediction work. That blurring is not merely cosmetic. It is the conditional mean made visible, the same pathology we examine under deterministic dynamics below, and it means a naive pixel-space model can look excellent on per-frame metrics while producing imaginations no planner could trust.

Pixel-space prediction remains valuable in two regimes. First, as a pretraining signal: predicting missing patches or future frames forces a network to absorb intuitive physics that transfers to downstream control. Here the pixel model is scaffolding rather than the final artifact: its weights seed an encoder that later serves a more abstract model, and the expensive pixel-level knowledge is distilled rather than queried. Second, as an interface: when humans must inspect or interact with the model's imagination, observations are the currency they understand. A research team diagnosing why a latent-world-model agent failed can gain more from watching ten decoded imagined frames than from reading a thousand latent vector norms.

Token-based models insert a discretization step between sensation and dynamics. A learned tokenizer, typically a vector-quantized variational autoencoder in the lineage of VQ-VAE, maps each observation to a short sequence of discrete codes drawn from a learned codebook. The world model is then a sequence model, frequently a transformer, over these code sequences, conditioned on actions. Conceptually, the tokenizer plays the role that quantization and symbol grounding played in classical perception pipelines, except that the vocabulary is discovered from data rather than designed by hand.

Discretization buys surprising advantages. Categorical distributions are easy for maximum-likelihood training to represent sharply, avoiding the blurry averages that plague Gaussian image models. Where a Gaussian decoder faced with two plausible futures must output their average, a categorical over tokens can place half its mass on each, keeping both futures crisp and recoverable. Sequence modeling over tokens inherits the mature toolkit of language modeling: stable scaling laws, teacher forcing, and architectures tuned over years of transformer development. Systems such as IRIS (Micheli et al., 2023) showed that a transformer over image tokens, combined with a policy over the same tokens, can reach strong sample efficiency on benchmark control tasks purely from pixels.

The costs mirror the benefits. Quantization is lossy in an uncontrolled way: the codebook decides what detail survives, and two perceptually distinct states may collapse to identical tokens, destroying distinctions a task needs. Because the codebook is learned before or alongside the dynamics, nothing in its training objective guarantees that the surviving distinctions are the ones your controller requires. Token boundaries also interact awkwardly with continuity: a smoothly moving object produces token churn that the dynamics model must smooth over. Finally, planning through a discrete imagination introduces gradient pathologies, since you cannot differentiate through a categorical sample without tricks. Search-based planners sidestep this by treating the token model as a simulator to query rather than a function to differentiate, which is one reason token-based world models pair naturally with sampling-based planning rather than gradient-based trajectory optimization.

Latent-state models, the workhorses of modern model-based reinforcement learning, learn a continuous encoding $z_t = e(o_{\le t})$ together with dynamics $p(z_{t+1} \mid z_t, a_t)$, training both jointly so that the latent space becomes useful for prediction. The World Models architecture of Ha and Schmidhuber (2018) established the template, a vision encoder, a recurrent dynamics model, and a controller trained inside the dream, and the Dreamer series (Hafner et al., 2020 onward) refined it into a family of state-of-the-art agents.

The central promise of latent models is that the network can discover the state variables that make dynamics simple. Recall from Chapter 2 that a good state renders the future conditionally independent of the past. A well-trained latent model approximates exactly that: its hidden state is a running belief about the aspects of the world that matter for predicting what comes next. Readers familiar with classical filtering will recognize this as the learned cousin of the Kalman filter's state estimate: in both cases the agent maintains a compressed summary of its observation history sufficient for forward prediction, and in both cases the quality of that summary determines everything downstream. The difference is that the Kalman filter's sufficiency was guaranteed by linear-Gaussian assumptions, while the latent model's sufficiency is only as good as its training losses and data coverage.

Because the space is continuous and differentiable, gradients flow through multi-step rollouts during training, and planners can optimize action sequences by gradient descent through the imagination. This differentiability is a structural advantage: it converts planning from a combinatorial search into a continuous optimization, which is dramatically cheaper when the imagination is trustworthy enough for gradients to point in useful directions.

The corresponding risk is that "useful for prediction" is measured against the training distribution and the training losses. A latent model can achieve excellent one-step prediction while quietly discarding information that a new task, a new reward, or a rare event will suddenly need. This is the problem of control-sufficient state, and it is central to state abstraction theory, which Part III develops formally. For now, hold onto the asymmetry: pixel models risk drowning in irrelevant detail, latent models risk starving relevant detail, and neither failure announces itself in the training loss. Both failures are invisible to the metric most teams watch first, which is why the evaluation practices of Part XI insist on testing representations against perturbations and tasks the training objective never mentioned.

Object-centric models bet that the right intermediate representation is neither pixels nor unstructured vectors but entities: persistent things with attributes (position, velocity, color) and relations (contact, containment, support). Slot-based architectures such as slot attention (Locatello et al., 2020) decompose a scene into a set of vectors, one per object, and dynamics models then predict how each slot evolves, often with interaction terms between slots. The bet has deep roots: physics engines, scene graphs in computer graphics, and relational calculi in classical AI all rest on the intuition that the world is naturally described as things undergoing local interactions, and that a learner given this description faces a vastly simpler induction problem.

The payoff is compositional generalization. If the model represents the world as $n$ objects obeying local interaction rules, then a scene with a novel arrangement of familiar objects, or even a slightly larger number of objects, remains in-distribution in a way that no flat representation guarantees. Physical reasoning, multi-object manipulation, and scene editing benefit visibly from this structure. Neural physics engines and the neuro-symbolic literature build directly on this bet. In classical terms, object-centric structure is an inductive bias purchased up front: it constrains the hypothesis class so drastically that far less data is needed, provided the constraint matches reality.

The difficulty is segmentation itself. Deciding what counts as an object is easy for billiard balls and brutal for deformables, fluids, granular media, and cluttered household scenes. Is a rope one object or infinitely many? Does a pile of sand have parts, and if so, where are their boundaries? Slot-based methods also impose a fixed slot count and struggle when objects appear, disappear, or merge. Object-centric modeling is therefore best understood as a strong prior that pays off enormously when its assumptions hold and misfires when they do not. The engineering pattern that works in practice is hybrid: let a slot-based module handle the rigid, well-segmented portion of the scene while a latent or pixel pathway absorbs everything the slots fail to capture.

At the abstract end of the spectrum sit symbolic models: states are discrete propositions or relational atoms (on(block_a, table)), and dynamics are rules, as in the planning languages classical AI used for decades. These models offer perfect compositionality, exact multi-step inference, and human-auditable structure. Their historical weakness was grounding: mapping raw perception to symbols reliably, and recovering when the mapping is wrong. A symbolic planner is only as good as its perceptual front end, and classical AI spent decades discovering that the last meter from pixels to predicates resists hand-engineering.

The contemporary picture is hybrid rather than adversarial. Learned perception proposes symbols; symbolic dynamics reasons over them; execution failures feed back to repair the grounding. Large language models have accelerated this trend by providing broad prior knowledge over symbols and plans. For world modeling specifically, symbolic structure tends to enter as a scaffold around a learned model, constraining its imagination to plausible relational configurations, rather than as the whole system. A typical modern stack might run a latent world model for continuous control while a symbolic layer tracks which high-level conditions hold, plans over long horizons where numeric precision is irrelevant, and flags when the learned model's imagination violates a relational invariant such as "a grasped object remains attached until released."

Table state-families summarizes the tradeoffs. Treat the rows as positions on a dial rather than mutually exclusive camps; production systems frequently combine several, for example a latent model whose decoder emits object slots.

Two cross-cutting remarks help you use this table well. First, the axis is really about where abstraction happens: nowhere (pixels), by a fixed pretrained module (tokens), jointly with dynamics (latents), by an inductive bias (objects), or by hand plus a grounding layer (symbols). Earlier abstraction means cheaper dynamics learning but greater risk of losing what matters. Second, the right position correlates strongly with how the model will be consumed, which is why we flagged the explicit/implicit distinction first. A model imagined over by a search procedure wants low-dimensional, differentiable latents. A model inspected by humans wants decodable outputs. A model serving a symbolic planner wants entities and relations.

Notice that these consumption patterns map onto the distinctions between predictive fidelity, representation quality, and decision usefulness drawn earlier. A pixel model maximizes predictive fidelity in the sense of matching observations, but its representation may support no compact reasoning. A latent model optimizes representation quality for the losses it was trained on, which may or may not align with decision usefulness for your task. Only by asking who consumes the model, and for what, can you decide which of these qualities to prioritize, and the table is best read as a menu of priorities rather than a leaderboard.

We will return to the question of what makes a latent state sufficient for control in Part III, and to the concrete architectures that implement each family in Part V.