Frontier Questions and Research Roadmaps › Experiment 2: Held-Out Combinations and When Factorization Fails

The second experiment asks whether a factorized representation can predict combinations of factors it never saw together. We construct a transition family in which the true interaction between two context factors is multiplicative, so that an additive model is structurally wrong, and then we add a stress condition in which even the multiplicative assumption fails.

The product law supplies a specific interaction that the additive feature family cannot represent exactly across the chosen factor levels. Even when an additive law is correct, its finite-data estimation and held-out predictions can still be tested. Here we use a normalized quadratic damping term, $-\kappa v|v|$, and let the coefficient depend on two context factors; this is a declared toy law, not a universal physical drag model. The capped regime asks a separate question: how does the product-feature model behave when that law changes? A test supporting a model is not thereby rigged, but testing a violated assumption makes the boundary of the result explicit.

The system is a scalar velocity with quadratic damping:

$$v_{t+1} = v_t + \Delta t \left( a_t - \kappa(s, p) \, v_t |v_t| \right) + \epsilon_t$$

where:

• $v_t$: scalar velocity at step $t$
• $a_t$: action (normalized acceleration) at step $t$
• $\Delta t = 0.2$: integration timestep
• $\kappa(s, p)$: damping coefficient determined by the surface factor $s$ and payload factor $p$
• $s \in \{1, 2, 3\}$: surface factor level
• $p \in \{1, 2, 3\}$: payload factor level
• $\epsilon_t$: additive noise at step $t$

where $s$ is a surface factor, $p$ is a payload factor, and both take levels in $\{1, 2, 3\}$. The coefficient $\kappa$ is what we will try to learn. In the main regime,

$$\kappa_{\text{main}}(s, p) = s \cdot p$$

In the stress regime, the damping saturates:

$$\kappa_{\text{stress}}(s, p) = \min(s \cdot p, \kappa_{\max}), \qquad \kappa_{\max} = 5$$

Here $\kappa_{\max}$ is the saturation ceiling: at high $s \cdot p$ the damping stops growing, so the true law is no longer a pure product.

The stress regime specifies a capped coefficient, $\kappa = \min(sp, 5)$. This is a deliberate alternative law, not a claim that every physical damping coefficient saturates. The held-out products are 3 and 6, so the test includes uncapped and capped combinations. A pure-product feature family is therefore misspecified in this regime. The coefficients and factors here are normalized toy quantities; the experiment does not establish a physical stability margin.

Rather than fit a full transition network, we estimate the single scalar $\kappa$ for each context by least squares, then study how meta-models map context to $\kappa$. This controls the feature families and makes the scalar identifiability condition explicit. It does not remove estimation error: the finite noisy identification samples produce uncertain coefficient labels, and that uncertainty propagates into the meta-model errors alongside approximation error. Known ground truth lets us measure those errors, not declare the identification step exact. A real system may also permit scalar fitting, or may require a richer model with additional measurement and model errors.

The identification samples $(v, a)$ pairs independently rather than rolling the system forward. For this one-parameter least-squares fit, a unique coefficient estimate requires $\sum_i (\Delta t\,v_i|v_i|)^2>0$. More spread can improve information under the stated noise model, but a visual velocity range is not a general persistent-excitation test. Gaussian velocity samples have unbounded support; the code imposes no hard $|v|\leq3$ limit.

Every factor level appears in training. What is held out is specific combinations: $(1,3)$, $(2,3)$, $(3,1)$, and $(3,2)$. The training set contains the diagonal $(1,1), (2,2), (3,3)$ and two symmetric off-diagonal cells $(1,2)$ and $(2,1)$.

The structure of the split is deliberate. Every level of $s$ and every level of $p$ appears in training, so the model has seen each factor individually. What it has not seen is the specific way high $s$ combines with high $p$ and vice versa. If the factors combine additively, the model can predict the held-out cells by extrapolating each factor's effect. If they combine multiplicatively, the model needs to know the product, and the training cells do not uniquely determine the product unless the model's functional form includes it. This is the sense in which the split tests the inductive bias rather than the data coverage.

Two meta-models map context to the damping coefficient:

• Factorized (additive) baseline. $\hat{\kappa} = b + w_s s + w_p p$. No interaction term.
• Interaction-capable baseline. $\hat{\kappa} = b + w_s s + w_p p + w_{sp} s p$. Characterizes the multiplicative structure.

Both are fit by least squares on the training cells. We check the design matrix rank and condition number so we are not fitting an arbitrary solution.

These diagnostic averages track the analytic coefficients closely in the unsaturated regime. Identification noise still enters every fitted meta-model, so held-out error combines that noise, estimation variance, and functional-form mismatch. The repeated study below measures this combined variability rather than attributing every error exclusively to structure.

We repeat the whole procedure eight times with fresh identification noise, so the error bars capture variation from sampling, identification and meta-model fitting under this fixed split. They are standard errors of each model's mean, not standard deviations of individual runs. Assess a model difference against uncertainty in its estimated mean difference; it need not exceed individual-run variability. The models share each repeat's training data, so paired repeat differences would provide a direct uncertainty estimate for their comparison. The separate marginal bars below do not supply that estimate.

In the multiplicative regime the interaction model has much lower held-out coefficient error than the additive model. The table also reports error against analytic truth on seen combinations: training-side approximation error is part of this comparison, not something to assume away. The product feature represents the specified law, while the additive family cannot represent it over the full grid. Under the capped law that advantage does not persist. These results concern two chosen feature families and splits, not every factorized representation or a universal compositionality advantage.

Under saturation, the interaction model's held-out error increases, while the additive model's error decreases relative to its own multiplicative-regime result. The mean ranking reverses in this seeded study. Both remain misspecified: neither functional form represents the capped product exactly. This is not evidence that saturation always favors an additive model. The product feature has lower error under the product law here, and that advantage does not survive this cap. Misspecification alone does not establish operational unreliability: deciding whether either error is acceptable requires a task-specific tolerance. Test the target regime and report the measured errors against that requirement.

The practical implications are direct:

• Test compositional claims with held-out combinations and report seen and unseen separately.
• When interpreting recovered factor effects, establish identifiability under the chosen feature family and report its design rank. Keep this identification claim separate from measured held-out performance.
• Include a violation condition so that the reader knows what happens when the assumed structure breaks.
• Prefer a model whose inductive bias matches the regime you will deploy in. Justify that regime from known structure or an appropriately powered target-system study; a small trial need not detect rare or later changes.

A predeclared held-out comparison can be a valid test even when it supports the proposed method. Adding a violation condition asks a separate question: how does performance change when the assumed law no longer holds? The capped regime supplies that boundary check here; it does not make all positive-only studies invalid. For deployment, test the target regime and ask whether the measured errors meet its requirements rather than seeking an unconditional model ranking.

An illustrative controlled example with a known ground-truth transition family, not a benchmark result.