Object-Centric and Relational Worlds

Consider a robot watching a kitchen table: a mug, a spoon, and a hand that picks up the spoon and stirs the mug. A conventional world model can compress each frame into one latent vector and learn a transition on those vectors. That can work well. In a typical stirring task, however, the hand and mug state are likely to matter more to the spoon's motion than countertop color. The useful causal structure is often local and compositional. An unconstrained flat latent can represent that structure, but it does not expose object correspondence, locality, or parameter sharing explicitly. Generalization to changed object counts, rearrangements, or exchanges is therefore possible, not guaranteed by the representation.

Object-centric and relational world models take the opposite bet: they represent the scene as a set of objects, each with its own state, connected by relations, and they factor the dynamics across those objects and relations. Instead of learning "what happens to the whole scene," they learn "what one object does to another." This chapter develops that bet in detail: how to discover objects without supervision, how to represent and reason about relations with graphs and message passing, how to keep object identity stable over time, and how interaction networks turn this structure into fast, generalizing dynamics models.

Throughout, keep the distinction from What Is a World Model? in mind: we separate what the model predicts (next object states, next relations, next masks) from how an agent uses those predictions (state estimation per object, planning actions that touch specific objects, imagining interventions). This separation matters more, not less, once structure enters the picture. A slot-based model can produce faithful-looking reconstructions while its slots fail to track objects through contact, which is a failure of the predictions the agent needs. Conversely, a model with mediocre reconstruction but accurate per-object dynamics can be excellent for control. We therefore keep four notions of model quality distinct as the chapter proceeds: predictive fidelity (does the rollout match reality), representation quality (do the slots correspond to objects the way we intend), uncertainty quality (does the model know where its predictions are unreliable), and decision usefulness (can an agent plan better with this model than with a flat one). Object structure is only worth its complexity if it improves prediction or decision-making; we will be precise about which.

In Sufficient State and State Abstraction we argued that a good state keeps everything decision-relevant and discards the rest. In Latent-Variable Representations we saw how autoencoders and variational methods can learn such states without direct state labels. But there is a third axis of representational choice: structure. A state can be sufficient and still use a flat, unstructured vector. Object-centric representations instead expose an explicit factorization: a set of entities, each with attributes, linked by relations. The distinction is not vector versus non-vector, since sets and graphs are stored in tensors too. It is the distinction between unstructured coordinates and an explicit object-and-relation interface.

The contrast is easiest to see with a scene of three balls. A distributed flat code need not assign fixed coordinates to each ball, and supervision or a suitable architecture can still make it systematic. Without an explicit object or set bias, however, consistent entity factorization is not guaranteed: the encoder and transition may have to learn correspondence, symmetry, and dynamics jointly. A structured representation supplies exchangeable containers and shared relational computations up front. It narrows the hypothesis class; it does not prevent distributed slot contents or eliminate permutation ambiguity.

The word "object" is doing a lot of work here, so let us pin it down.

That last clause matters. Objects are not just visual segments. A pair of scissors is one object or two depending on whether you care about its hinge. A cloud can be an object for a weather model and not for a manipulation model. For many prediction and control tasks, a useful abstraction suppresses internal degrees of freedom irrelevant to the current question. Once you know where a rigid mug is and how it is moving, molecular coordinates are unnecessary for predicting its short-term pose. Object-centric representations are a conditional bet that such a factorization matches the environment and task; when it does not, the bias can hurt.

Notice how this definition depends on the task, and therefore on the agent. The decomposition of the world into objects is not written into the pixels; it is a hypothesis about which partition of the scene yields the simplest dynamics for the questions being asked. A billiards model wants balls. A cloth-folding model wants regions of fabric with internal state. A traffic model wants vehicles and maybe platoons. This task-relativity is not a defect of the definition but a feature: it tells us that object discovery should ideally be evaluated by the dynamics and decisions it enables, not by how well the discovered segments match human segmentations. We return to that evaluation question at the end of the chapter.

The cognitive science literature calls the underlying puzzle the binding problem: perception must group features that belong to the same thing and maintain those groupings over motion, deformation, and occlusion. Images contain useful boundary and motion cues, but not explicit persistent identities or a unique correct partition. Spelke's core-knowledge studies and Baillargeon's looking-time experiments tested infants of particular ages with controlled possible and impossible events. Those results are commonly interpreted as evidence for early expectations of persistence, solidity, and continuity; they do not by themselves establish the mechanism or a mature object concept. Object-centric models borrow the design lesson cautiously: a matched inductive bias can improve learning, while a mismatched one reduces flexibility.

Before the machinery, it is worth being explicit about what the structured bet purchases:

• Sample efficiency. A factored dynamics model shares parameters across objects that use the same dynamics family and feature semantics. A learned rule for one ball can then be reused for another. A generic flat transition can learn the same locality, but it does not receive locality or object-wise sharing as an architectural guarantee.
• Compositional generalization. A pairwise model can be applied to compatible graphs with more objects or new arrangements. By contrast, a fixed-width joint-state model without padding, masks, or equivariant modules needs an interface change for a different object count. This is a claim about that restricted interface, not an impossibility result for every vector representation.
• Interpretability and interventions. "If you push the red block" maps directly to one node and its edges. A flat latent can support the same query through a learned probe or readout, but it does not provide an explicit object handle without that additional interface.
• Grounded actions and diagnosis. When perception preserves identity and the planner accepts object-indexed actions, structured states can support commands such as "grasp the cup" and can localize a rollout error to one object. These are design motivations whose benefit depends on the task, action parameterization, and reliability of the representation.

The cost is real, too. Object-centric models force hard commitments: how many objects, what counts as one, what the per-object state should contain. Each commitment is a place where the inductive bias can be wrong. A mismatch can propagate errors through the model, although whether it is worse than an unstructured alternative depends on data, optimization, and the task. The rest of this chapter is about making those commitments well, and about what breaks when they are wrong.

The first problem is discovery. Without segmentation supervision, the model must learn a useful scene decomposition. Reconstruction alone does not require human-aligned objects: parts, textures, and background regions can also minimize the loss. A widely studied family uses slot representations: the scene is represented by a fixed set of $K$ exchangeable latent vectors, with architectural competition intended to encourage different slots to specialize.

Competition is one mechanism for making slots specialize; it is not a definition of every slot model. Without competition, redundancy is possible, although other priors, routing rules, or objectives can also break symmetry. The next two subsections show how soft masks and attention normalization create competition for explanatory weight over pixels or tokens.

Several early unsupervised object-discovery methods use component-wise scene decomposition with exclusive or competitive masks: MONet (Burgess and colleagues, 2019), IODINE (Greff and colleagues, 2019), GENESIS (Engelcke and colleagues, 2020), and GENESIS-V2 (Engelcke and colleagues, 2021). Their inference mechanisms differ: MONet uses recurrent attention, IODINE iteratively refines variational posterior parameters, GENESIS uses an autoregressive scene prior, and GENESIS-V2 uses embedding clustering with stochastic stick breaking. In the following common mixture view, slot $k$ decodes to a mask logit $m_k(u)$ and appearance prediction $a_k(u)$:

where:

• $\hat{x}(u)$: the reconstructed color at pixel location $u$
• $a_k(u)$: slot $k$'s appearance prediction at pixel $u$
• $m_k(u)$: slot $k$'s mask logit at pixel $u$
• $\gamma_k(u)$: the normalized mask weight at pixel $u$, a softmax over slots that says what fraction of pixel $u$ slot $k$ explains
• $k'$: an index running over all $K$ slots

The softmax gives each pixel one unit of mixture weight divided fractionally among components. This creates competition and can encourage near-exclusive masks, but it does not force a one-hot or semantically correct object assignment. Decoding a mask from one latent supplies a spatial-coherence bias rather than enforcing connected contours. IODINE performs iterative amortized variational inference using likelihood, gradients, masks, and other refinement inputs; it is inspired by iterative refinement rather than a literal alternating mask/appearance algorithm. On simple multi-object datasets, these models can learn object-aligned components without segmentation labels, while split, merged, texture-based, and background components remain possible.

Slot Attention (Locatello and colleagues, 2020) is an influential object-centric encoder. It resembles transformer attention, but combines competition across slots with a second normalization that forms a weighted mean for each slot.

We have a set of input feature vectors $x_1, \dots, x_N$ (for an image, one per spatial location, typically from a CNN) and $K$ slots $s_1, \dots, s_K$. Compute dot-product affinities between slots and inputs:

where:

• $s_k$: the current vector of slot $k$
• $x_n$: the $n$-th input feature vector
• $W_q$, $W_k$: learned projection matrices applied to the slots (queries) and the inputs (keys)
• $d$: the projected dimension; when query and key coordinates have roughly fixed, weakly correlated variance, dividing by $\sqrt{d}$ stabilizes the typical dot-product scale
• $d_{kn}$: the raw affinity between slot $k$ and input $n$

The first normalization is the competitive one: normalize each input's affinities over slots. A second normalization over inputs will form each slot's weighted mean below.

where:

• $a_{kn}$: the weight that input $n$ assigns to slot $k$
• $k'$: an index running over all $K$ slots, so the weights on slots sum to one for each input

In standard attention, each query usually normalizes over keys. Here, each input distributes one unit of weight across slots. Comparable affinities split that weight; with two equal logits, the split is exactly one half each. This soft allocation creates competition and encourages a partition, but does not force a discrete or object-correct assignment.

The competition is easiest to see in a toy matrix of affinities.

Each column sums to one, so increasing one slot's share reduces the others' shares. The middle columns show the greatest ambiguity between the two slots.

Canonical Slot Attention next renormalizes each slot's competitive weights across inputs to form a weighted mean, then updates the slot. With a small $\epsilon>0$ for stability,

where:

• $a_{kn}$: the competitive weight that input $n$ assigns to slot $k$
• $w_{kn}$: slot $k$'s weights renormalized across inputs, which sum to one over $n$
• $W_v$: a learned value projection applied to the inputs
• $u_k$: the weighted-mean value update for slot $k$
• GRU: a gated recurrent unit that blends the new evidence $u_k$ into the slot's previous content
• MLP and LayerNorm: a small multilayer perceptron applied after layer normalization, added as a residual update

The GRU in the middle of this update is easy to overlook but does real work: because the update is applied iteratively, the slot needs a way to keep what it has already learned about its object while incorporating new evidence, rather than overwriting itself at every pass. The GRU's gating provides exactly that selective memory within the iteration loop.

The canonical setup samples initial slots from learned Gaussians and uses three refinement iterations, although the count is a tunable hyperparameter. The process resembles soft clustering: attention weights act like assignments and slots like iteratively updated representatives. After the configured iterations, many variants decode each slot independently with a spatial-broadcast decoder and combine masks with a softmax mixture. Independent decoding is a strong component-separation bias, not a requirement; SLATE instead uses a joint transformer decoder conditioned on all slots.

Two properties make slot attention attractive for world modeling:

• Permutation symmetry. When initialization and downstream processing preserve the symmetry, permuting initial slots permutes the outputs. No discovery-time index is privileged, which discourages fixed conventions such as "slot three is always leftmost" but does not prevent swaps or every index-based degeneracy.
• Variable input sets, fixed object capacity. Attention can process a variable number of input features, but a fixed $K$ is a direct ceiling on simultaneously represented objects. Slot width, iteration count, positional encodings, and encoder resolution are additional capacity and resolution choices.

For world modeling we need object structure in time, not just per frame. A per-frame encoder returns an unordered set, so correspondence across frames is unresolved. SAVi, or conditional Slot Attention (Kipf and colleagues, 2022), propagates a fixed set of prior slots through a predictor and corrects them with the current frame. This learned temporal conditioning replaces an explicit matching stage and encourages continuity, but it does not guarantee a stable track ID or solve births and deaths. SAVi used optical flow as a training target, treating coherent motion as a useful grouping cue rather than a definition of objecthood. SAVi++ (Elsayed and colleagues, 2022) replaces flow supervision with depth prediction and evaluates synthetic depth and sparse-LiDAR supervision.

SLATE (Singh, Deng, and Ahn, 2022) uses a dVAE to discretize image patches, maps the token embeddings into continuous slots with Slot Attention, and autoregressively reconstructs the token grid with a transformer conditioned jointly on all slots. The discrete vocabulary belongs to image patches, not to the slots, and SLATE itself is an image model rather than a temporal world model. SlotFormer (Wu and colleagues, 2023) supplies the separate temporal step by predicting sequences of continuous slots and evaluating them for video prediction and model-based control.

Reconstruction quality alone is an incomplete proxy for object-centric segmentation or downstream usefulness. A model can reconstruct well while splitting one object or merging touching objects, because the pixel mixture can be correct even when the decomposition is not the one the task needs. Evaluation can therefore add adjusted Rand index (ARI) against ground-truth masks, foreground ARI (FG-ARI), and downstream tests of dynamics prediction or control. Each metric answers a different question.

Once the scene is a set of objects, the natural language for how they influence each other is a graph. Objects are nodes; relations, known or learned, are edges. Graph representations give us a principled way to write dynamics that depend on which objects interact, not just on the global scene. The move from sets to graphs is the move from "there are things" to "these things affect each other in these ways," and it is the second half of the structured bet.

Formally, a graph-structured scene is a tuple $(\mathcal{V}, \mathcal{E}, u)$: nodes $\mathcal{V}$ with per-object states $h_i$ (position, velocity, pose, appearance summary), edges $\mathcal{E}$ with per-relation attributes $e_{ij}$ (distance, contact type, spring rest length, relative velocity), and optionally a global attribute $u$ describing scene-level context (gravity, table friction). The three levels of the tuple are worth keeping separate in your head, because different physical regularities live at different levels: forces between objects live on edges, intrinsic object behavior and damping live on nodes, and uniform fields and shared environment properties live in the global attribute. Battaglia and colleagues (2018) made the influential argument that this "graph network" block generalizes a large family of models, from interaction networks to transformers, and that the choice of graph is a statement of relational inductive bias: an assumption about which interactions can matter.

The bias is a commitment. A fully connected graph assumes any object may affect any other; a sparse spatial graph assumes locality. Getting this wrong can waste computation or omit relevant interactions. With $n$ objects, a complete graph has $O(n^2)$ edge activations, so memory and compute grow quadratically even though a shared edge function keeps parameter count fixed. Sparse graphs are useful when their construction covers the interaction support.

The quadratic cost of the dense choice is easy to state and easy to see.

At forty objects the dense graph carries nearly twenty times the edges of this ideal 2-regular graph. This analytic line is not the exact count of the union-symmetrized $k$-nearest-neighbor builder used later: that data-dependent construction can contain between $nk/2$ and $nk$ undirected edges depending on reciprocal choices.

For systems whose direct interactions are local in space, a $k$-nearest-neighbor or radius graph can be a cheap approximation when its edges cover the true interaction support. The risk is explicit: $k$-nearest neighbors can omit contacts in dense regions or add irrelevant neighbors in sparse ones, while fixed-radius graphs omit long-range effects and cannot express mediated influence in one hop.

Many graph-based dynamics models use a variant of one update, applied one or more times. The recipe has three stages: compute a message per edge, aggregate messages per node, and update each node's state. For every edge from sender $j$ to receiver $i$, an edge function computes a message:

Each node aggregates its incoming messages with a permutation-invariant reduction $\square$ (sum, mean, max, or set attention with shared order-independent scoring):

and a node function produces the next node state:

where:

• $h_j$, $h_i$: the current states of sender node $j$ and receiver node $i$
• $e_{ji}$: the attributes of the edge from sender $j$ to receiver $i$
• $\phi_e$: the learned edge function, shared across all edges
• $m_{ji}$: the message computed on the edge from sender $j$ to receiver $i$
• $\mathcal{N}(i)$: the set of nodes with an edge into node $i$, the in-neighborhood of $i$
• $\square$: a permutation-invariant reduction, including sum, mean, max, or order-independent set attention
• $\bar{m}_i$: node $i$'s aggregated incoming message
• $\phi_v$: the learned node function, shared across all nodes
• $h_i'$: the next state of node $i$, one message-passing round later

A global update $\phi_u$ over aggregated node and edge summaries can weave in scene-level context. Three properties of this recipe carry the real weight:

• Permutation invariance of aggregation. Sum, mean, and max do not depend on neighbor order, so the same parameters can be applied to variable-size compatible graphs. Sum preserves the vector total: six equal, co-directed forces yield six times one force, while general force vectors can differ or cancel. Mean instead normalizes by degree.
• Parameter sharing across edges and nodes. One edge function is applied everywhere. A learned rule can transfer across pairs when relation types, feature semantics, and physical regimes match.
• Composition. After $L$ synchronous local rounds, information can travel up to $L$ hops. Dependence across a path requires at least its graph distance in rounds unless a global solver or nonlocal edge supplies a shortcut.

Transformer self-attention can also be viewed as message passing on a complete graph with attention-weighted aggregation. Key differences include initialization, normalization, iterative recurrent slot updates, exchangeability, and the decoder and training objective. Slot Attention allocates each input's unit weight across slots, creating competition that encourages a partition; standard attention normally lets each query normalize independently over keys. We develop the transformer side in Transformers and Neural State-Space Architectures.

Finally, a symmetry note. Absolute position can often be replaced by relative position $\Delta x=x_j-x_i$ and relative velocity $\Delta v=v_j-v_i$, while retaining necessary invariant attributes such as mass, type, and spring rest length. When all inputs and operations respect translation symmetry and there are no position-dependent fields or boundaries, this makes translation invariance automatic. Rotation-equivariant architectures such as the equivariant graph networks of Satorras and colleagues (2021) further restrict the hypothesis class and can improve sample efficiency when that symmetry is valid on the evaluated task.

Graph machinery becomes a world model when it predicts next object states and rolls them forward. Filtering, planning, and error attribution can use that structured state after their interfaces are adapted to sets, graphs, and uncertain correspondence. The Interaction Network (Battaglia and colleagues, 2016) is an influential template used by many later relational models.

Given per-object states $x_i$ (say, position and velocity) and relations $r_{ji}$, an interaction network computes:

1. An effect per relation: $\;\varepsilon_{ji} = \phi_e(x_j, x_i, r_{ji})$, a learned summary of how $j$ pushes on $i$.
2. An aggregated effect per object: $\;\bar{m}_i = \sum_{j \in \mathcal{N}(i)} \varepsilon_{ji}$.
3. A dynamics output per object: $\;\Delta_i = \phi_v(x_i, \bar{m}_i)$, interpreted as the object's acceleration or state derivative.

For the pairwise spring and n-body systems considered here, this mirrors an important physical decomposition: pair effects add vectorially before a node update. The architecture supplies that relational computation template while the MLPs still learn the message and update functions. Numerical integration then updates velocities and positions. When connectivity is state-dependent, as in radius or $k$-nearest-neighbor graphs, recomputing edges lets topology change with the state; fixed string or mesh relations need not be rebuilt. Battaglia and colleagues (2016) evaluated n-body systems, rigid-body collisions, and non-rigid strings, including domain-specific tests at object counts different from training. Those results demonstrate qualitative count generalization within the same physical domain, not a universal two-or-three-to-ten result.

The Visual Interaction Network (Watters et al., 2017) closed the perception loop: a visual encoder mapped short frame windows to per-object latent state codes, an interaction-network dynamics predictor rolled those codes forward, and a linear state decoder output per-object position and velocity. This is structurally object-centric: encode visual observations into object states, predict in structured latent space, and decode task-relevant states. Here the latent space is a set of objects with relations rather than one vector, and the transition is factored across that set.

The recipes above assume the edge structure is given. Neural Relational Inference (Kipf and colleagues, 2018) instead infers discrete latent interaction types variationally from trajectories. In the charged-particle experiment, the relevant types distinguish attraction from repulsion; separate experiments include non-interacting edges. The encoder amortizes inference for new trajectories, and training uses a differentiable relaxation or surrogate for the discrete choices. A diffuse posterior can indicate ambiguity under a calibrated, adequate model, but it can also reflect misspecification, optimization failure, label symmetry, or poor calibration. A planner should treat it as epistemic evidence only after those conditions are checked.

A common alternative in named simulators is to define edges geometrically: connect $k$ nearest neighbors or all objects within radius $r$. For state-dependent graphs the edges are recomputed, and the network learns per-edge effects. This embeds locality and avoids latent-edge inference, but interactions omitted by the rule are invisible in that message-passing step.

Later work scaled the template to larger particle and mesh simulations. Graph Network Simulator (Sanchez-Gonzalez and colleagues, 2020) predicts per-particle accelerations for fluids and granular or deformable materials using fixed-radius connectivity. MeshGraphNets (Pfaff and colleagues, 2021) applies message passing to deformable surfaces and aerodynamics on computational meshes. These papers demonstrate learned simulation on their reported benchmarks; they do not establish general solver parity at a measured fraction of engineering cost.

• Noisy one-step training. GNS trains a one-step acceleration predictor and adds random-walk noise to input trajectories to expose the model to perturbed states resembling rollout errors. It does not train on short self-generated rollouts in the cited method.
• Encode-process-decode. GNS and MeshGraphNets encode node and edge features, apply several message-passing rounds, and decode dynamics outputs. A useful design heuristic is to ensure the processor receives the information required for relational reasoning; encoder width alone does not determine whether the graph carries relational information.

Contrastive Learning of Structured World Models (C-SWM, Kipf and colleagues, 2019) learned object-factored latent states with graph-structured transitions and demonstrated planning in its reported abstract environments. SlotFormer (Wu and colleagues, 2023) learned transformer dynamics over slots and evaluated video prediction and model-based control. In such systems, stable slots plus an object-indexed planner can enable object-specific action selection and error analysis. A flat latent can support those queries through an additional learned interface, but it does not expose object handles directly.

Discovery gives you objects now. A world model needs objects over time: the mug you plan to grab next is the mug you saw two seconds ago, even though it passed behind the kettle. This section covers the three intertwined problems of temporal binding. They are worth separating because they fail in different ways and are solved by different machinery: permanence is about memory and prediction through gaps in observation, tracking is about assigning identities across frames, and the sufficiency of the tracked state determines whether any of it is useful for decisions.

Object permanence is continued existence while an object is unobserved; an expectation of a continuous hidden trajectory is an additional assumption. In controlled violation-of-expectation studies, infants are shown matched possible and impossible events, and longer looking at the impossible event is commonly interpreted as evidence of an expectation under the experiment's controls. IntPhys (Riochet and colleagues, 2018) and ADEPT (Smith and colleagues, 2019) adapt surprise-based physical probes to models. CLEVRER (Yi and colleagues, 2020) is different: it evaluates descriptive, explanatory, predictive, and counterfactual question answering over collision videos. Matched occlusion probes are designed to reduce surface shortcuts and test whether surprise depends on representing a hidden object; forgetting alone does not logically imply zero surprise under every classifier or statistical model.

Permanence requires memory and prediction without direct observations of the object. A transition prior can advance hidden state while indirect evidence from occluder boundaries, contacts, camera motion, or other objects still corrects it. Transition error can make a reappearing observation harder to associate, especially when appearance cues are weak, but it need not force failure. Flat and slotted latents can both preserve hidden-object beliefs. Slots make a per-object trajectory easier to inspect; they do not certify permanence. Training on documented synthetic occlusions or explicit object-permanence objectives can provide relevant gradients; the unsupported generic "slot dropout reconstructs masked slots" recipe is not assumed here.

Tracking is the problem of assigning object identities across frames. In tracking-by-detection, filtering and data association are two central subproblems, alongside detection, gating, and track management:

• Per-object filtering. A tracker maintains a state estimate or belief for each track and predicts it forward before correction. A Kalman tracker carries a Gaussian estimate; other trackers need not represent a full posterior.
• Data association. Given predicted and detected states, the Hungarian algorithm finds a minimum-cost one-to-one assignment for the supplied cost matrix. Dummy assignments permit unmatched tracks or detections; lifecycle rules then interpret them as misses, candidate births, or deaths.

SORT (Bewley and colleagues, 2016) combines a constant-velocity Kalman model with Hungarian association over detections. DeepSORT (Wojke and colleagues, 2017) adds an appearance metric to help association when geometry is ambiguous. Their explicit assignments are inspectable, although overlap, missed detections, and long disappearances can still cause swaps or failed re-identification.

Slot-based video models instead encourage continuity by recurrent conditioning. In SAVi, prior slots are predicted forward and corrected using the current frame, removing a separately exposed matching decision but not guaranteeing identity. A swap may be silent unless the system adds identity-consistency diagnostics. Explicit association is easier to inspect; recurrent conditioning is end-to-end. Either design should be validated under the task's occlusion, contact, and re-entry conditions rather than chosen by a binary rule.

It is worth connecting this to sufficient state. The necessary contents depend on the task: velocity when motion prediction needs it, grasp-relevant geometry for manipulation, and appearance when re-identification is required. A slot is only as useful as the sufficiency of what it encodes. A slot that binds an object cleanly but lacks pose may support video prediction while failing a grasp planner. One design is to keep the slot free-form while supervising or probing particular variables. In the code section, the synthetic world gives us ground-truth positions and velocities directly.

Let's make message passing concrete with the smallest possible physical system: two unit masses connected by one spring with stiffness $k = 6$ and rest length $L_0 = 1$, moving along a line with damping $c = 0.6$. Small enough to do by hand, rich enough to exercise every stage of the recipe: an edge function, an aggregation, a node update, and an integration step. At some instant, with $x_i$ the position of mass $i$ and $v_i$ its velocity:

The worked physical system has one undirected relation. We will display its equal-and-opposite forces as two directed effects. That representation allows both effects to be carried, but a generic learned directed-edge MLP does not enforce Newton's third law unless antisymmetry is built into the parameterization or one pair force is applied with opposite signs.

Edge messages. The signed scalar spring law is $k(r-L_0)$; its magnitude is $k|r-L_0|$, with direction supplied separately. Here $r=1.2$, so the stretch is $0.2$ and the magnitude is $1.2$. The physical force on mass 1 points toward mass 2, so $\varepsilon_{2\to1}=+1.2$, while the paired force is $\varepsilon_{1\to2}=-1.2$. A learned edge function can use relative position and rest length to approximate this vector law without absolute positions. Exact action-reaction symmetry still requires an antisymmetric construction; otherwise data-sparse regimes are one risk among several sources of approximation error.

Aggregation. Each mass has exactly one neighbor, so $\bar{m}_1=1.2$ and $\bar{m}_2=-1.2$. In a larger lattice, aggregation forms the vector sum of attached spring forces. For two equal, co-directed contributions, averaging would return half of the correct sum; in general, directions and magnitudes differ and may cancel.

Node update. The node function combines the aggregated force with damping: $a_i = \bar{m}_i - c\,v_i$, where $a_i$ is the acceleration of mass $i$ (with unit masses, net force and acceleration are numerically equal). With $v = 0.3$ for both masses:

One semi-implicit Euler step with $\Delta t = 0.02$, updating velocity first with $v_i' = v_i + \Delta t\, a_i$ and then position with $x_i' = x_i + \Delta t\, v_i'$, gives $v_1' = 0.3 + 0.02 \times 1.02 = 0.3204$ and $v_2' = 0.3 + 0.02 \times (-1.38) = 0.2724$. The separation becomes

which is down from $1.2$: the stretched spring is pulling the masses together, with damping bleeding energy.

The edge function in this example is a straight line in the separation, which makes its job easy to picture.

This curve is one one-dimensional, fixed-parameter slice of the learned relation. The two-dimensional model below receives relative position, relative velocity, and rest length and must learn a direction-dependent vector message across that larger input space.

For a fixed-size joint state $(x_1,x_2,v_1,v_2)$, a generic MLP can learn relative differences and symmetry, but does not obtain them by construction. A different object count changes a fixed-width interface unless the design adds padding, masks, sharing, or set-equivariant modules. Message passing supplies shared local computation and permutation-compatible aggregation directly. The following code tests that inductive bias against an edge-free per-object baseline.

We will build a small interacting-particle world, train an interaction network to predict its dynamics, measure how rollout error compounds, and test the central claim of the chapter: that a relational model trained on four-object scenes generalizes to larger graphs where a flat per-object model does not. We close with a minimal data-association tracker to make identity concrete. Throughout, our "slots" are the true object states, positions and velocities, so that questions about representation quality do not contaminate the questions about dynamics and identity we intend to isolate.

The world is a set of point masses in the plane connected by springs. Dynamics are semi-implicit Euler: accelerations come from spring forces along active edges plus global damping.

The model follows the recipe exactly. The edge function consumes relative position, relative velocity, and rest length; aggregation is a sum via index_add_; the node function maps own velocity plus aggregated messages to a predicted acceleration. Note that all edge features are relative, so translation invariance is built in rather than learned: shift the whole scene by any vector and every edge input is unchanged, hence so is every prediction.

In the seeded run below, the final acceleration loss is about 21 times lower for the interaction network. The representational advantage is architectural: an independent node lacks pair context. The measured finite-training gap still depends on capacity, optimization, minibatches, and training duration.

Now the distinction that this book keeps returning to: one-step accuracy is cheap, closed-loop rollout is the real test. We roll each trained model out from the initial state of every test trajectory, feeding its own predictions back as inputs, and record position error per horizon.

The panels show both transfer and its limit. On this held-out 6-node graph, the rollout retains broad motion directions but several paths differ visibly in extent and curvature by 40 steps. The architecture provides separate places for relational messages and node updates, but this plot does not identify damping with one module or spring rhythm with the other. Shared size-independent functions are an architectural explanation for transfer across the evaluated counts, not proof that the learned modules recovered a unique physical decomposition.

Finally, consider two noisy point tracks that cross while observation order is reshuffled each frame. We maintain a last-displacement constant-velocity prediction and solve a two-by-two Euclidean-cost assignment with the Hungarian algorithm. This is a SORT-like filtering-plus-assignment toy, not the SORT algorithm itself, which uses a Kalman state for detected boxes and an overlap-based association cost. Near the crossing, position-only current-frame association is highly uncertain under the stated equal-appearance noise model.

Prediction helps away from the crossing by supplying where each track should be if its identity persists. Here it is insufficient at closest approach: noisy last-displacement estimates drive repeated swaps. Appearance features, calibrated Kalman uncertainty, or multi-frame smoothing could add evidence. Recurrent slot conditioning changes where continuity enters the system, but it can also swap unless identity consistency is checked explicitly.

The key parameters for the interaction network and its training setup are:

• SPRING_K: Spring stiffness of the synthetic world. It sets the magnitude of the pairwise force the edge function must learn from relative positions and rest lengths.
• DAMPING: Global velocity damping applied in the node update. It is directly predictable from a node's own velocity, unlike configuration-specific spring forces.
• DT: Integration step for semi-implicit Euler. For a fixed physical horizon and within the method's stable regime, smaller values usually reduce discretization error but require more steps.
• STEPS: Number of rollout steps per trajectory. It fixes the horizon over which compounding rollout error is measured.
• hidden: Width of the hidden layers in the edge and node MLPs. Larger values give the message and update functions more capacity at a higher training cost.
• BATCH_SIZE: Number of transition items sampled per gradient step. Larger batches often reduce gradient variance at higher memory cost.
• TRAIN_STEPS: Number of optimizer updates applied to each model with the shared minibatch schedule.
• lr: Adam optimizer step size. Larger values can accelerate early progress but also increase instability or overshoot.

Object-centric and relational world models introduce failure modes alongside their benefits. This section separates those limitations from the results the cited methods have demonstrated.

The failure modes begin at discovery.

Discovery is not solved. Synthetic benchmarks with simple objects demonstrate the mechanism, while clutter, occlusion, deformation, and domain shift require separate evaluation rather than a blanket "works best" ranking. A fixed slot count $K$ is a hard capacity ceiling; empirical models can split, merge, or allocate slots to background, but the exact failure pattern depends on data and objective. More fundamentally, "object" is task-relative. Reconstruction pressure does not know whether a manipulation policy needs mug-and-handle structure or whether a traffic model needs vehicles and platoons. Pixel fidelity can therefore disagree with decision usefulness.

Occlusion and contact are important, challenging cases for explicit association and recurrent slot continuity. An undetected swap can attach predictions to the wrong history and may mislead an object-indexed planner; confidence and downstream failure are not automatic. Long occlusions also require transition predictions to remain useful without direct object observations. Factorization can localize error to a slot, which may help diagnosis or planning when downstream components use that structure, but it does not make rollouts self-correcting.

Several cited simulators fix connectivity with a radius rule or mesh, while NRI-style methods learn latent relations. Dense pairwise latent-edge inference has $O(n^2)$ candidate pairs, and richer edge-type posteriors add statistical and computational burden. An imperfect fixed topology can be harmful when omitted or spurious edges cannot be compensated by the learned functions, but a message function may also learn to suppress irrelevant edges. A fixed topology does not represent uncertainty over graph structure directly, so topology error is harder to diagnose, though residuals, uncertainty, or learned edge weights can still reveal it.

Compositional generalization is bounded by what the representation and training identify. Interaction Networks report domain-specific transfer to different object counts within the same physical family. That does not establish zero-shot transfer to a qualitatively new interaction law. A shared untyped edge function chiefly supplies an architectural route to count and configuration generalization; typed relations, relation embeddings, meta-learning, or additional training are possible routes across interaction kinds. The architecture alone guarantees none of these extrapolations. Part IV develops related geometry, physical dynamics, and causal tools.

The cited work demonstrates learned simulation of fluids, granular materials, deformable surfaces, and aerodynamics in research benchmarks, alongside object-factored planning in the reported C-SWM and SlotFormer environments. IntPhys and ADEPT make learned physical expectations testable without claiming historical priority or hard-coded permanence, solidity, and continuity. The defensible lesson is conditional: encoding a correct structural bias can improve data efficiency and generalization, while mismatch, finite capacity, data, and optimization determine the outcome. The next chapter applies the object-centric lens to the agent's own state, and Part IV adds geometry, physics, and causal structure.

• Objects are task-relative abstractions. Explicit sets and graphs expose entity correspondence and relations but do not make every decomposition correct.
• Slot mixtures and Slot Attention create soft competition. Canonical Slot Attention normalizes across slots, then renormalizes over inputs to form each slot's weighted-mean update. Neither step guarantees object-aligned binding.
• Message passing shares edge and node functions across compatible graphs. Relative features provide translation invariance only when every remaining input and operation respects that symmetry.
• The experiment compares a relational network with a size-flexible independent-node baseline. It supports transfer across the evaluated 4-, 6-, and 8-node graphs, while its selected trajectory still drifts visibly.
• Tracking combines prediction and association, but the seeded crossing example records identity swaps. Recurrent slot conditioning also encourages rather than guarantees continuity.
• Fixed capacity, topology mismatch, occlusion, and new interaction laws remain separate failure modes. Structured bias helps when it matches the task and can hurt when it does not.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about object-centric and relational world models.