Object-Centric and Relational Worlds › Why Objects? From Flat Latents to Structured Scenes

In Sufficient State and State Abstraction we argued that a good state keeps everything decision-relevant and discards the rest. In Latent-Variable Representations we saw how autoencoders and variational methods can learn such states without direct state labels. But there is a third axis of representational choice: structure. A state can be sufficient and still use a flat, unstructured vector. Object-centric representations instead expose an explicit factorization: a set of entities, each with attributes, linked by relations. The distinction is not vector versus non-vector, since sets and graphs are stored in tensors too. It is the distinction between unstructured coordinates and an explicit object-and-relation interface.

The contrast is easiest to see with a scene of three balls. A distributed flat code need not assign fixed coordinates to each ball, and supervision or a suitable architecture can still make it systematic. Without an explicit object or set bias, however, consistent entity factorization is not guaranteed: the encoder and transition may have to learn correspondence, symmetry, and dynamics jointly. A structured representation supplies exchangeable containers and shared relational computations up front. It narrows the hypothesis class; it does not prevent distributed slot contents or eliminate permutation ambiguity.

The word "object" is doing a lot of work here, so let us pin it down.

That last clause matters. Objects are not just visual segments. A pair of scissors is one object or two depending on whether you care about its hinge. A cloud can be an object for a weather model and not for a manipulation model. For many prediction and control tasks, a useful abstraction suppresses internal degrees of freedom irrelevant to the current question. Once you know where a rigid mug is and how it is moving, molecular coordinates are unnecessary for predicting its short-term pose. Object-centric representations are a conditional bet that such a factorization matches the environment and task; when it does not, the bias can hurt.

Notice how this definition depends on the task, and therefore on the agent. The decomposition of the world into objects is not written into the pixels; it is a hypothesis about which partition of the scene yields the simplest dynamics for the questions being asked. A billiards model wants balls. A cloth-folding model wants regions of fabric with internal state. A traffic model wants vehicles and maybe platoons. This task-relativity is not a defect of the definition but a feature: it tells us that object discovery should ideally be evaluated by the dynamics and decisions it enables, not by how well the discovered segments match human segmentations. We return to that evaluation question at the end of the chapter.

The cognitive science literature calls the underlying puzzle the binding problem: perception must group features that belong to the same thing and maintain those groupings over motion, deformation, and occlusion. Images contain useful boundary and motion cues, but not explicit persistent identities or a unique correct partition. Spelke's core-knowledge studies and Baillargeon's looking-time experiments tested infants of particular ages with controlled possible and impossible events. Those results are commonly interpreted as evidence for early expectations of persistence, solidity, and continuity; they do not by themselves establish the mechanism or a mature object concept. Object-centric models borrow the design lesson cautiously: a matched inductive bias can improve learning, while a mismatched one reduces flexibility.

Before the machinery, it is worth being explicit about what the structured bet purchases:

• Sample efficiency. A factored dynamics model shares parameters across objects that use the same dynamics family and feature semantics. A learned rule for one ball can then be reused for another. A generic flat transition can learn the same locality, but it does not receive locality or object-wise sharing as an architectural guarantee.
• Compositional generalization. A pairwise model can be applied to compatible graphs with more objects or new arrangements. By contrast, a fixed-width joint-state model without padding, masks, or equivariant modules needs an interface change for a different object count. This is a claim about that restricted interface, not an impossibility result for every vector representation.
• Interpretability and interventions. "If you push the red block" maps directly to one node and its edges. A flat latent can support the same query through a learned probe or readout, but it does not provide an explicit object handle without that additional interface.
• Grounded actions and diagnosis. When perception preserves identity and the planner accepts object-indexed actions, structured states can support commands such as "grasp the cup" and can localize a rollout error to one object. These are design motivations whose benefit depends on the task, action parameterization, and reliability of the representation.

The cost is real, too. Object-centric models force hard commitments: how many objects, what counts as one, what the per-object state should contain. Each commitment is a place where the inductive bias can be wrong. A mismatch can propagate errors through the model, although whether it is worse than an unstructured alternative depends on data, optimization, and the task. The rest of this chapter is about making those commitments well, and about what breaks when they are wrong.