Object-Centric and Relational Worlds › Limitations and Impact

Object-centric and relational world models introduce failure modes alongside their benefits. This section separates those limitations from the results the cited methods have demonstrated.

The failure modes begin at discovery.

Discovery is not solved. Synthetic benchmarks with simple objects demonstrate the mechanism, while clutter, occlusion, deformation, and domain shift require separate evaluation rather than a blanket "works best" ranking. A fixed slot count $K$ is a hard capacity ceiling; empirical models can split, merge, or allocate slots to background, but the exact failure pattern depends on data and objective. More fundamentally, "object" is task-relative. Reconstruction pressure does not know whether a manipulation policy needs mug-and-handle structure or whether a traffic model needs vehicles and platoons. Pixel fidelity can therefore disagree with decision usefulness.

Occlusion and contact are important, challenging cases for explicit association and recurrent slot continuity. An undetected swap can attach predictions to the wrong history and may mislead an object-indexed planner; confidence and downstream failure are not automatic. Long occlusions also require transition predictions to remain useful without direct object observations. Factorization can localize error to a slot, which may help diagnosis or planning when downstream components use that structure, but it does not make rollouts self-correcting.

Several cited simulators fix connectivity with a radius rule or mesh, while NRI-style methods learn latent relations. Dense pairwise latent-edge inference has $O(n^2)$ candidate pairs, and richer edge-type posteriors add statistical and computational burden. An imperfect fixed topology can be harmful when omitted or spurious edges cannot be compensated by the learned functions, but a message function may also learn to suppress irrelevant edges. A fixed topology does not represent uncertainty over graph structure directly, so topology error is harder to diagnose, though residuals, uncertainty, or learned edge weights can still reveal it.

Compositional generalization is bounded by what the representation and training identify. Interaction Networks report domain-specific transfer to different object counts within the same physical family. That does not establish zero-shot transfer to a qualitatively new interaction law. A shared untyped edge function chiefly supplies an architectural route to count and configuration generalization; typed relations, relation embeddings, meta-learning, or additional training are possible routes across interaction kinds. The architecture alone guarantees none of these extrapolations. Part IV develops related geometry, physical dynamics, and causal tools.

The cited work demonstrates learned simulation of fluids, granular materials, deformable surfaces, and aerodynamics in research benchmarks, alongside object-factored planning in the reported C-SWM and SlotFormer environments. IntPhys and ADEPT make learned physical expectations testable without claiming historical priority or hard-coded permanence, solidity, and continuity. The defensible lesson is conditional: encoding a correct structural bias can improve data efficiency and generalization, while mismatch, finite capacity, data, and optimization determine the outcome. The next chapter applies the object-centric lens to the agent's own state, and Part IV adds geometry, physics, and causal structure.