Compositionality and Systematic Generalization › Measuring Systematic Generalization

If we care about compositional generalization, we must evaluate for it. In factorial datasets with multiple observations per world, an observation-level i.i.d. split will usually put the same world combinations in both training and test, so it does not isolate novel composition. The chance of overlap depends on the number of observations per cell and the split fraction, rather than following from i.i.d. sampling alone. A model can score well on the repeated combinations while remaining unconstrained on combinations withheld at the world level. Everything in this section exists to close that gap.

The canonical recipe has three requirements.

• Primitive coverage: every primitive (mechanism value, object type, relation) should appear in multiple training combinations. Marginal appearance is necessary but not sufficient: verify state-action support, excitation, sample adequacy, and component estimation quality in the conditions relevant to the held-out pairing. Only after those controls does failure specifically implicate factorization or composition rather than primitive learning.
• Held-out combinations: test worlds are combinations absent from training. A clean way to guarantee this is to hold out a structured subset, such as an entire anti-diagonal of a factorial design, so that every factor value remains well represented in training. The anti-diagonal has a pleasing property: removing it removes every combination of the form $(m, c-m)$ while leaving every row and every column of the grid populated, so both mechanisms keep full coverage of their value sets.

The following figure shows the split used in the worked example: removing the anti-diagonal leaves every row and column populated.

• Multiple seeds and splits: with few held-out cells, results are noisy. Repeat the split with different held-out cells and report the spread. A model that passes the anti-diagonal split might fail when a different diagonal is held out, and that sensitivity is itself a finding: it suggests the model's composition is only partially factorized, with some mechanisms entangled in ways other splits do not expose.

In the compositional generalization literature for language and reasoning, splits are often scored with divergence measures that quantify how different train and test compositions are. The same discipline applies here: design the split so memorizing observed combinations is insufficient. For each held-out world, check not only whether every mechanism value appears in training, but whether the data cover and excite the component behavior at relevant states and actions. These controls make composition the intended novelty; they cannot prove it is the only source of difficulty.

A single test loss hides the distinctions that matter. At minimum, report:

• One-step error on held-out combinations: this measures whether the composed predictor works on sampled true states without rollout compounding. Good one-step error with poor autoregressive performance points toward compounding or rollout-distribution shift, but additional state-coverage checks are needed before declaring the composition globally correct.
• Fixed-action autoregressive rollout error as a function of horizon: replay the same exogenous actions while feeding model predictions back as states. This isolates multi-step predictive compounding; it is not closed-loop control. The curve's shape also matters: roughly linear growth suggests persistent bias, while rapid growth suggests instability, subject to checking scale, dynamics, and noise.
• Planning success with the model held fixed: train the model on the compositional split, then let a planner use it in held-out worlds. This is the decision-usefulness number, and it can decouple from prediction accuracy when the model's errors are systematically biased in directions the planner exploits. Zero-mean errors under a fixed sampling distribution do not by themselves make a model serviceable for planning: a maximizing planner can select optimistic tails, especially when errors are heteroscedastic or correlated across candidates. Lower average error can likewise coexist with a harmful directional bias, so decision usefulness must be measured directly.
• Sample efficiency for adaptation: given a small budget of trajectories in a new world, how quickly does fine-tuning recover accuracy? Selective updates and fast repair are evidence consistent with localized mechanisms, but generic feature reuse or meta-learning can produce the same pattern. Track which parameters change and control support, optimizer, and adaptation budget before attributing the result to factorization.

No single number among these substitutes for the others, and reporting all four is cheap relative to the cost of deploying a model whose evaluation summarized away the failure mode that matters.

Several established benchmarks probe related abilities, but their split semantics differ. SCAN tests compositional command generalization through structured command splits. CLEVR-CoGenT tests visual question answering under held-out combinations of object colors and shapes. COGS uses structured generalization splits for semantic parsing. Procgen instead evaluates transfer to held-out procedurally generated levels, while manipulation suites can be designed around held-out object combinations. The shared lesson is to define the intended shift explicitly; only some of these benchmarks implement the exact fixed-primitives, held-out-combinations design used in this chapter.

Many controlled domains admit an approximate factorial construction once candidate mechanism families are explicit. Where interventions are feasible, enumerate or sample combinations, verify primitive support in training, hold out a structured subset, and run the metric suite above. Continuous, coupled, irreversible, or ethically nonmanipulable mechanisms may require partial factorial designs or different stress tests. A domain-specific split is useful when its assumptions match the system.

A particularly informative diagnostic is the swap test: take environments that differ in a candidate mechanism, swap that mechanism, and check whether predictions change accordingly. Passing is evidence consistent with the proposed factorization, while the categorical-ID baseline below has no sharing rule that requires it to pass. Coincidence, support differences, and unequal task difficulty remain alternative explanations, so use matched controls and repeated swaps.

The swap test deserves a precise statement. Let worlds $w = (m_1, m_2)$ and $w' = (m_1', m_2')$ share no mechanism values. Check both cross-combinations $(m_1, m_2')$ and $(m_1', m_2)$. An asymmetric failure can suggest which candidate mechanism or routing path is problematic, but support, noise, and unequal difficulty must be ruled out before causal localization. Simulation permits exact swaps; real systems may permit controlled physical recombination.