Compositionality and Systematic Generalization › Why Compositionality Matters for World Models

Recall from Part I that a world model earns its keep by supporting decisions, not just by fitting observed transitions. This distinction becomes sharp when we think about generalization. A model can have excellent one-step prediction accuracy on data drawn from the training distribution and still fail in closed-loop use when a policy reaches states it has not seen. Compositionality is one structural response to this problem when the relevant mechanisms can be reused, and it is worth understanding precisely what that response does and does not guarantee.

The mechanism of the failure is a feedback loop between the agent and the model. During training, the states in the dataset follow the trajectories of whatever policy or process collected them. When a planner searches over action sequences, it may favor states where the predicted outcome is good, including states near or beyond the regions the training policy visited. Without support constraints or adequate uncertainty controls, out-of-distribution behavior can therefore become a central operating regime and the planner can exploit optimistic model errors. Constrained or well-covered planning need not create this shift.

The natural world is combinatorial. A kitchen is not one monolithic scene; it is a particular mug, on a particular counter, with a particular drawer state, combined from a vocabulary of objects, surfaces, and mechanisms. If your vocabulary has $K$ factors with $q$ options each, the number of distinct worlds is $q^K$. Under any feasible data budget, you will observe only a tiny subset of these combinations. A model that treats each observed world as an independent learning problem has no principled prediction for an unseen combination. A model that represents the world as a composition of separately learned parts can determine that prediction from the behaviors of the parts, provided those parts were sufficiently observed and the composition rule is valid.

The exponential arithmetic deserves a moment of reflection, because it is the entire motivation for the chapter. With even a modest vocabulary, say twenty factors with five options each, the space of possible worlds contains $5^{20}$, roughly ninety-five trillion, distinct configurations. That finite space could be enumerated in principle, but exhaustive coverage is computationally and statistically impractical: even a passive sampler with full support would need an enormous coupon-collector budget to see every configuration. The practical route is to learn at the level of the factors themselves, so that each piece of data can update reusable components rather than only one point in the grid. This is an engineering response to exponential configuration growth, not a mathematical impossibility theorem about finite datasets.

The figure below makes the counting argument concrete: it plots the size of the configuration space against the fraction any realistic dataset can cover.

It is worth separating two things that are easy to conflate. Prediction quality asks whether the model's simulated rollouts match reality. Decision usefulness asks whether an agent planning against those rollouts chooses good actions. Valid compositional structure can improve prediction on novel combinations when the component estimates have relevant support and the composition rule is correct. It does not by itself make unsupported search trustworthy: calibration, uncertainty handling, and robust decision procedures remain separate requirements. We will keep both lenses in view throughout.

It is also worth pulling apart two further notions that are often collapsed into "the model is good." Representation quality asks whether the model's internal variables correspond to the real factors of variation in the world, so that the pieces it can recombine are the pieces the world recombines. Uncertainty quality asks whether the model knows what it does not know, so that its confident predictions in novel combinations are earned rather than reflexive. A model can predict well without having the right representations (a lookup table predicts perfectly on seen worlds), and it can have the right representations without calibrated uncertainty (a factorized model can be confidently wrong about a mechanism it saw in narrow conditions). Compositionality, taken seriously, addresses prediction and representation together, but it does not by itself deliver calibrated uncertainty; that requires the machinery from Stochasticity and Uncertainty. Keeping these four notions separate, predictive fidelity, representation quality, uncertainty quality, and decision usefulness, prevents the most common evaluation mistake, which is reading a low test loss as evidence of all four at once.