Compositionality and Systematic Generalization › Recombining Objects, Relations, and Skills

Factorized mechanisms tell us that the world decomposes. Object-centric and relational representations, which we developed in Part III, tell us what the pieces usually are. This section connects the two.

Physical environments are naturally described as a set of objects, each with its own state, its own intrinsic dynamics, and interactions governed by relations. When a world model represents state as a set of object states rather than a flat vector, it inherits a powerful form of combinatorial structure:

• Permutation sharing: all objects of the same type share the same dynamics. A model that learns "how mugs slide" once can apply it to every mug, which is data amplification exactly like the factorized case. Every mug in every training scene becomes a training example for the shared mug-dynamics module, so the effective dataset for that module is the total number of mug-steps across all scenes, not the number of scenes.
• Permutation invariance: predictions should not depend on the arbitrary order of objects in the input. Architectures that process sets or graphs enforce this by construction. Without architectural invariance or adequate order augmentation, a model trained with mugs always listed before pans can fail when the order reverses; a non-invariant architecture may still learn the symmetry from sufficiently varied data.
• Variable cardinality: a model that computes per-object and pairwise quantities can, in principle, run on a scene with more objects than it ever saw during training, because nothing in its computation depends on the count. The per-object encoder applies to each object regardless of how many there are, and the aggregation combines any number of pairwise messages into a fixed-size output.

These three properties make the intended sharing explicit in object-centric models. A flat-vector model can also implement equivariance, parameter sharing, locality, or algebraic factorization, but a flat model without equivalent structure has no architectural guarantee of those symmetries. An object-centric model predicts object-wise futures and relation changes, then assembles the next state subject to local couplings such as collisions rather than treating object futures as independent.

Relations are the second half of the vocabulary. Contact, support, containment, and collision apply to pairs of objects. Implementing a relation as a shared function, as message-passing architectures do, creates a route for transfer across pairs. A learned "supported-by" relation can transfer from mugs and shelves to books and tables only when its inputs capture the relevant physics and the new pair lies within the relation model's supported regime.

The caveat in that last clause deserves emphasis. A message-passing network can implement "shared relation" at different scopes. A relation function based on physical pair quantities such as relative position, velocity, and contact geometry can potentially transfer broadly. A module conditioned on type embeddings may still share parameters, but only within the transfer scope supported by those types. If it memorizes individual object pairs, it cannot reuse the relation on a new pair. Whether the learned sharing matches the intended mechanism is an empirical question, and the swap test later in this chapter is designed to probe it.

This is where the relational inductive bias becomes concrete. A model with per-object encoders, pairwise relation functions, and permutation-invariant aggregation encodes sharing directly, while a fixed-length flat network without equivalent sharing leaves recombination unconstrained. The component inventory can scale with objects and relation types rather than scene combinations, but empirical coverage still depends on support and estimation quality. We will see graph simulators and slot-based models in Part V; the point here is that architecture can make a composition rule explicit.

A distinct generalization axis deserves its own name. Productivity is the ability to handle structures longer, deeper, or larger than those seen in training: a chain of ten dominoes after training on three, a sentence with a nested clause after training on short ones, a robot assembly with twelve parts after training on five. Size-agnostic shared computation can make such inputs architecturally executable without adding a new module for every length or object count. That does not guarantee accurate extrapolation, and a fixed-size or otherwise monolithic model can also be designed for variable length. Productivity is where the limits of learned mechanisms show up: a contact model that is approximately right per interaction can be badly wrong about how errors accumulate over a long chain.

The domino example makes both sides of this vivid. Applying the same local "one domino topples the next" mechanism ten times is trivial for a compositional model; it is literally the same computation repeated, and no architectural change is needed to go from three dominoes to ten. But ten applications also means ten chances for the local mechanism's error to compound, and if each toppling slightly overestimates the transferred momentum, the tenth domino's behavior may be badly wrong even though every two-domino prediction was excellent. Productivity therefore splits into two questions that must be evaluated separately: can the model run on larger structures (an architectural question, where compositional models win by construction), and are its predictions still accurate on larger structures (a statistical question about error accumulation, where no architecture saves you automatically). The evaluation methodology later in the chapter, particularly horizon-resolved rollout error, is exactly the instrument needed for the second question.