Object-Centric and Relational Worlds › Object-Centric and Relational Worlds

Consider a robot watching a kitchen table: a mug, a spoon, and a hand that picks up the spoon and stirs the mug. A conventional world model can compress each frame into one latent vector and learn a transition on those vectors. That can work well. In a typical stirring task, however, the hand and mug state are likely to matter more to the spoon's motion than countertop color. The useful causal structure is often local and compositional. An unconstrained flat latent can represent that structure, but it does not expose object correspondence, locality, or parameter sharing explicitly. Generalization to changed object counts, rearrangements, or exchanges is therefore possible, not guaranteed by the representation.

Object-centric and relational world models take the opposite bet: they represent the scene as a set of objects, each with its own state, connected by relations, and they factor the dynamics across those objects and relations. Instead of learning "what happens to the whole scene," they learn "what one object does to another." This chapter develops that bet in detail: how to discover objects without supervision, how to represent and reason about relations with graphs and message passing, how to keep object identity stable over time, and how interaction networks turn this structure into fast, generalizing dynamics models.

Throughout, keep the distinction from What Is a World Model? in mind: we separate what the model predicts (next object states, next relations, next masks) from how an agent uses those predictions (state estimation per object, planning actions that touch specific objects, imagining interventions). This separation matters more, not less, once structure enters the picture. A slot-based model can produce faithful-looking reconstructions while its slots fail to track objects through contact, which is a failure of the predictions the agent needs. Conversely, a model with mediocre reconstruction but accurate per-object dynamics can be excellent for control. We therefore keep four notions of model quality distinct as the chapter proceeds: predictive fidelity (does the rollout match reality), representation quality (do the slots correspond to objects the way we intend), uncertainty quality (does the model know where its predictions are unreliable), and decision usefulness (can an agent plan better with this model than with a flat one). Object structure is only worth its complexity if it improves prediction or decision-making; we will be precise about which.