Object-Centric and Relational Worlds › Interaction Networks and Compositional Dynamics

Graph machinery becomes a world model when it predicts next object states and rolls them forward. Filtering, planning, and error attribution can use that structured state after their interfaces are adapted to sets, graphs, and uncertain correspondence. The Interaction Network (Battaglia and colleagues, 2016) is an influential template used by many later relational models.

Given per-object states $x_i$ (say, position and velocity) and relations $r_{ji}$, an interaction network computes:

1. An effect per relation: $\;\varepsilon_{ji} = \phi_e(x_j, x_i, r_{ji})$, a learned summary of how $j$ pushes on $i$.
2. An aggregated effect per object: $\;\bar{m}_i = \sum_{j \in \mathcal{N}(i)} \varepsilon_{ji}$.
3. A dynamics output per object: $\;\Delta_i = \phi_v(x_i, \bar{m}_i)$, interpreted as the object's acceleration or state derivative.

For the pairwise spring and n-body systems considered here, this mirrors an important physical decomposition: pair effects add vectorially before a node update. The architecture supplies that relational computation template while the MLPs still learn the message and update functions. Numerical integration then updates velocities and positions. When connectivity is state-dependent, as in radius or $k$-nearest-neighbor graphs, recomputing edges lets topology change with the state; fixed string or mesh relations need not be rebuilt. Battaglia and colleagues (2016) evaluated n-body systems, rigid-body collisions, and non-rigid strings, including domain-specific tests at object counts different from training. Those results demonstrate qualitative count generalization within the same physical domain, not a universal two-or-three-to-ten result.

The Visual Interaction Network (Watters et al., 2017) closed the perception loop: a visual encoder mapped short frame windows to per-object latent state codes, an interaction-network dynamics predictor rolled those codes forward, and a linear state decoder output per-object position and velocity. This is structurally object-centric: encode visual observations into object states, predict in structured latent space, and decode task-relevant states. Here the latent space is a set of objects with relations rather than one vector, and the transition is factored across that set.

The recipes above assume the edge structure is given. Neural Relational Inference (Kipf and colleagues, 2018) instead infers discrete latent interaction types variationally from trajectories. In the charged-particle experiment, the relevant types distinguish attraction from repulsion; separate experiments include non-interacting edges. The encoder amortizes inference for new trajectories, and training uses a differentiable relaxation or surrogate for the discrete choices. A diffuse posterior can indicate ambiguity under a calibrated, adequate model, but it can also reflect misspecification, optimization failure, label symmetry, or poor calibration. A planner should treat it as epistemic evidence only after those conditions are checked.

A common alternative in named simulators is to define edges geometrically: connect $k$ nearest neighbors or all objects within radius $r$. For state-dependent graphs the edges are recomputed, and the network learns per-edge effects. This embeds locality and avoids latent-edge inference, but interactions omitted by the rule are invisible in that message-passing step.

Later work scaled the template to larger particle and mesh simulations. Graph Network Simulator (Sanchez-Gonzalez and colleagues, 2020) predicts per-particle accelerations for fluids and granular or deformable materials using fixed-radius connectivity. MeshGraphNets (Pfaff and colleagues, 2021) applies message passing to deformable surfaces and aerodynamics on computational meshes. These papers demonstrate learned simulation on their reported benchmarks; they do not establish general solver parity at a measured fraction of engineering cost.

• Noisy one-step training. GNS trains a one-step acceleration predictor and adds random-walk noise to input trajectories to expose the model to perturbed states resembling rollout errors. It does not train on short self-generated rollouts in the cited method.
• Encode-process-decode. GNS and MeshGraphNets encode node and edge features, apply several message-passing rounds, and decode dynamics outputs. A useful design heuristic is to ensure the processor receives the information required for relational reasoning; encoder width alone does not determine whether the graph carries relational information.

Contrastive Learning of Structured World Models (C-SWM, Kipf and colleagues, 2019) learned object-factored latent states with graph-structured transitions and demonstrated planning in its reported abstract environments. SlotFormer (Wu and colleagues, 2023) learned transformer dynamics over slots and evaluated video prediction and model-based control. In such systems, stable slots plus an object-indexed planner can enable object-specific action selection and error analysis. A flat latent can support those queries through an additional learned interface, but it does not expose object handles directly.