Compositionality and Systematic Generalization › Limitations and Impact

The honest reading of this chapter is that compositional generalization is purchased with assumptions, and each assumption can fail. The ICM principle says mechanisms are separable, but real mechanisms interact: a gripper's contact dynamics depend on both the object's surface and the gripper's material in ways that are not cleanly additive, and in fluids or soft bodies the very notion of a separable mechanism can be approximate at best. The factor labels in our experiment were handed to the model. From raw observations alone, those labels are underdetermined in general; identifying them requires additional assumptions or information, which may come from interventions, temporal or auxiliary structure, grouped observations, supervision, or architectural constraints. A model can also factorize along the wrong axes, such as visual appearance rather than physical mechanism. These failures need not trigger high uncertainty, which is why swap tests, mechanism-level diagnostics, and uncertainty quantification remain necessary even for structured models.

The wrong-axis failure deserves emphasis. A model that factors by appearance can pass splits dominated by visual variation yet fail physical swaps, such as visually similar surfaces with different friction. The structural factorization alone need not expose that confusion, although a calibrated uncertainty method may. Design diagnostics around physical interventions rather than visual variation, and test for invariance to appearance together with sensitivity to dynamics.

There is also a standing tension with scale. Strong structural priors can improve sample efficiency and compositional coverage when their commitments are correct, while diverse data can expose a less structured model to many useful recombinations. These approaches are complements rather than guaranteed substitutes: architecture constrains how components can combine, and data determines which behavior has empirical support. The data-side implications, including interaction, curation, and synthetic data, are taken up in Part VI.

An implicit factorization provides no architectural certificate for which recombinations are grounded, yet it remains empirically diagnosable through held-out-combination tests, swaps, probes, and interventions. Explicit structure makes its commitments easier to inspect and test, at the cost of committing to a factorization that may be wrong. In safety-relevant systems, that inspectability can be useful only when paired with evidence that the structure is accurate, calibrated uncertainty, and robust decision procedures; inspectability alone is not a safety guarantee.

The compositional viewpoint has produced useful tools in concrete systems, but the scope of each result matters. Interaction Networks, for example, reported transfer to some synthetic physical systems with object counts beyond training; that result does not establish arbitrary arrangement or real-world transfer. A correctly specified mechanism family can pool informative interactions across related systems when the data excite the relevant parameters. Option models can let a planner reason over temporally extended actions rather than only primitive actions, but whether this improves planning depends on model accuracy, initiation and termination semantics, and search. Primitive-covered splits, horizon-resolved rollout error, and planning-success measures test those claims in the situations where an agent will act.