Compositionality and Systematic Generalization › Factorized Mechanisms

The formal backbone of compositionality in world models is the idea that the world's dynamics decompose into modules that can vary independently. This section develops that idea formally and explains what it buys you.

Building on Causal Models and Interventions, the central assumption is this: the process that generates the world is not one big function but a set of mechanisms, each responsible for a distinct aspect of the dynamics, and changing one mechanism does not change the others.

Before formalizing, it helps to see the intuition behind the assumption. In the kitchen example, changing the countertop material need not change the drawer hinges, and replacing a hinge need not change countertop friction. When such autonomy holds, an intervention can isolate one mechanism while the others remain stable. The formal principle below treats that locality as a modeling hypothesis rather than a universal law of nature, and it also tells us what would falsify the hypothesis: a change in one proposed mechanism that systematically alters another.

The phrase "replacing it with a different mechanism of the same kind" is doing real work. It says that mechanism families are shared across worlds: there is a family of possible friction behaviors, and worlds differ in which member of that family they instantiate. This is what makes cross-world comparison possible at all, and it is what the mechanism-indexed formulation below will make explicit. In Haavelmo's econometric program, autonomy is a modeling requirement to test over specified perturbations: a proposed structural relation should remain stable under the interventions for which it is claimed to be structural, rather than being assumed invariant under every possible change.

Concretely, suppose the state decomposes or factors into components, and the transition can be written as a product of conditional mechanisms:

$$p(s_{t+1} \mid s_t, a_t) \;=\; \prod_{k=1}^{K} p_k\big(s_{t+1}^{(k)} \;\big|\; \mathrm{pa}(k)\big),$$

where:

• $s_t$: the current state at time $t$, with $a_t$ the action taken at that time
• $s_{t+1}^{(k)}$: the $k$-th component of the next state, the slice of the world that mechanism $k$ is responsible for updating
• $p_k$: the conditional distribution that mechanism $k$ assigns to its own component
• $\mathrm{pa}(k)$: the (hopefully small) set of variables that mechanism $k$ depends on, drawn from the current state, the action, and possibly earlier components of $s_{t+1}$ in an acyclic same-slice graph
• $K$: the number of mechanisms (and corresponding state components) into which the dynamics are factorized
• $\prod_{k=1}^{K}$: the product over all $K$ mechanisms, which reassembles the full next-state distribution from the per-mechanism pieces

Read the product form carefully, because every term in it carries a commitment. As written, it defines a dynamic Bayesian network only when any dependencies among next-state components form a directed acyclic graph; the factors can then be arranged in a topological order, and the product is a normalized joint distribution with the corresponding local conditional independences. A next-state component may therefore depend on an earlier next-state component in that order. Genuinely simultaneous or bidirectional constraints, such as a coupled collision equilibrium, require different semantics: for example, structural equations with a well-defined equilibrium or normalized potentials with a partition function. Multiplying arbitrary cyclic conditionals is not enough to define a joint distribution.

Two properties make this useful:

• Sparsity: each mechanism depends on few variables, which reduces input dimension and can lower estimation burden; the local function may still be complex or poorly identified.
• Modularity: each mechanism can be reparameterized or intervened on without touching the others, which is exactly what lets you recombine them.

The two properties offer complementary benefits. Sparsity can reduce the dimensionality and estimation burden of each mechanism. Modularity lets the same estimated mechanism be reused across worlds. A sparse but world-specific mechanism still requires relearning per world; a dense mechanism can nevertheless remain compositional if it is autonomous, shared, and combined by a valid rule. Sparsity helps statistical and computational efficiency, but it is not logically required for recombination.

In a world with multiple environments, we can go further and index each mechanism by a setting. Write a world as a tuple $w = (m_1, m_2, \ldots, m_K)$, where each $m_k \in \mathcal{M}_k$ is a value of the $k$-th mechanism (friction is low or high, the object is a mug or a pan). The transition kernel is then:

$$p_w(s' \mid s, a) \;=\; p(\,s' \mid s, a;\, m_1, \ldots, m_K\,),$$

where:

• $p_w$: the transition kernel belonging to world $w$
• $s'$: the next state
• $s, a$: the current state and the action taken
• $m_1, \ldots, m_K$: the mechanism settings, held fixed for the given world; conditioning on them selects one member of each mechanism family, so each world has its own transition kernel

The set of possible worlds is the combinatorial product $\mathcal{M}_1 \times \cdots \times \mathcal{M}_K$. This is the formal statement of the kitchen intuition: worlds are combinations of mechanisms, and the number of worlds grows exponentially in the number of factors.

Notice what this formulation does and does not assume. It assumes worlds differ only in mechanism settings, not in the structure of the factorization itself; the same $K$ families exist in every world. When a new world contains a genuinely new kind of mechanism, no element of $\mathcal{M}_1 \times \cdots \times \mathcal{M}_K$ describes it, and we land in the "novel structure" category of the taxonomy later in this chapter. The formulation also quietly assumes that the mechanism settings are stable within a world, which fits controlled transfer settings but not worlds whose mechanisms change over time; nonstationarity requires the machinery of later chapters.

The payoff is statistical. Compare two learning strategies in a world with $K$ binary mechanisms, giving $2^K$ possible worlds, where you collect $N$ trajectories per world.

A monolithic model treats each world as a separate learning problem. It must estimate $2^K$ distinct transition functions, seeing $N$ trajectories each, for a total of $N \cdot 2^K$ trajectories before it can predict everywhere. Worse, nothing in its hypothesis space tells it what to do with a world it has never seen. Even if the model is a flexible neural network conditioned on a world identifier, the identifier is an arbitrary label: knowing the transition for world $0$ and world $1$ constrains nothing about world $2$, because the label carries no information about which mechanisms differ. The monolithic model's knowledge is a set of isolated islands, with no bridges between them.

A factorized model can update a mechanism value from every trajectory in which it is active. In this full-factorial thought experiment, mechanism $k$ with a given value appears in $N \cdot 2^{K-1}$ trajectories (all worlds sharing that value), which is $2^{K-1}$ times the raw trajectory count available to one isolated world model. With $K = 10$ binary mechanisms, that is 512 times as many pooled trajectories containing the value. This is not a 512-fold sample-efficiency theorem: the full factorial collection itself contains $N \cdot 2^K$ trajectories, and under a fixed balanced budget $B$, a binary value appears in roughly $B/2$ trajectories regardless of $K$. The structural saving is instead that the model stores a library of $K \cdot q$ value-specific components plus a composition rule rather than a separate component for every world.

It is worth seeing where the pooling number comes from, because the same counting discipline governs experimental design. Fix a value, say $m_k = 0$, and count the worlds that contain it: exactly half of all $2^K$ worlds, that is $2^{K-1}$ of them. Each of those worlds contributes $N$ trajectories, so the value occurs in $N \cdot 2^{K-1}$ trajectories. Whether those trajectories provide proportionally more independent information depends on excitation, correlation, and model specification. Under a fixed total budget, the useful advantage is that each trajectory informs several reusable components and that the design can balance coverage of mechanism values without enumerating every world.

The amplification factor is easiest to absorb as a curve: for each number of binary mechanisms, compare the trajectories available per mechanism value under the two strategies.

This construction echoes a recurring idea in system identification: choose or learn a dynamical model structure, then estimate its behavior from input-output data. Textbook examples often fit a few interpretable parameters such as stiffness or damping, while the broader field also includes structure selection and flexible nonlinear models. Factorized world models add an explicit reusable decomposition: component functions may be learned flexibly, but the architecture still encodes which parts are separable and how they combine.

The spring example captures the useful analogy. A spring constant identified on one test rig is intended to describe the spring rather than the rig, so it can transfer when the relevant operating assumptions continue to hold. A learned factorized component aims for the same scene-independent semantics, but that semantics is a hypothesis to validate rather than a consequence of its name. Modern methods may learn both the component function and parts of its parameterization from data.

There is an important caveat that separates textbook treatments from practice. Everything above assumed the model receives the mechanism identity $(m_1, \ldots, m_K)$ as an input. In real problems, the factors are not labeled. The agent observes pixels and forces, and must infer which latent mechanisms are present and how they combine.

This is the domain of causal representation learning, and it is hard. As discussed in Latent Variable Representations and Object-Centric and Relational Worlds, learning a factorized representation from passive observations alone is impossible in general without additional inductive biases, as formalized by Locatello et al. (2019): many different latent factorizations can produce identical observations. Interventions are one powerful way to break this symmetry by changing one thing while holding others fixed. Temporal or auxiliary structure, grouped observations, supervision, and architectural assumptions can also provide identifying information under their own conditions.

The impossibility claim is worth stating in intuitive terms so that it does not read as pessimism about a solvable problem. A passive dataset is a set of samples from one joint distribution, and without additional assumptions that distribution can support many latent descriptions: an invertible transformation may leave observed data unchanged while scrambling which latent controls what. If friction and wetness always vary together, passive samples alone do not distinguish them. An intervention that varies friction while holding wetness fixed can cut that tie, while other auxiliary information can do so in different problem settings. The hierarchy below grades problems by how much identifying structure is available.

The practical hierarchy to keep in mind:

• Oracle factors: mechanism identities are given (simulators, labeled domains, controlled experiments). This removes the factor-discovery problem, but architecture, state-action support, excitation, finite-sample estimation, optimization, misspecification, and validation remain real concerns.
• Inferred factors from known variables: the factors exist in the observation space (object positions, temperatures) and the model must learn to route them. Methods from Object-Centric and Relational Worlds live here. The hard part shifts from discovering the factors to deciding which observed variables each mechanism reads and writes, and mistakes in routing show up as silent composition failures.
• Fully latent factors: the model must discover both the factors and the mechanisms from raw interaction. This remains an active research problem. Weak architectural biases such as permutation invariance or locality and deliberately structured data collection are two sources of identifying information, but neither combination is a general recipe and each method must state its assumptions.

The hierarchy is useful because it separates statistical, architectural, and representational difficulties. Moving down it generally requires more identifying assumptions or auxiliary information; interventions are one important source, but not the only one. A team that knows where it stands can budget data collection and validation accordingly.

The same factorization logic applies one level up. In the options framework of Sutton, Precup, and Singh (1999), an option is a temporally extended closed-loop policy defined by an initiation set, an internal policy, and a termination condition. An option model or skill-effect model separately summarizes the option's state-transition and reward consequences for planning. If you learn such a model for each skill, for example how "push object" changes position or how "open container" changes container state, then planning can compose those abstract consequences. We will develop the hierarchical machinery for this in Part V's chapter on hierarchical and hybrid models and in Part VII on planning.

The parallel is useful but not an identity. A low-level mechanism maps state and action variables to next-state variables; an option model maps a starting state through a temporally extended policy to a distribution over termination states, durations, and rewards. The option's initiation set determines where it may start, while its termination rule determines when it hands control back. Both settings replace "learn what happens in each full situation" with reusable pieces plus a combination rule, but the temporal abstraction adds policy and termination semantics that a one-step mechanism does not have.