State, Physics, Causality, and Memory Evaluation › Physical Rule and Constraint Tests

Michael BrenndoerferAugust 2, 202611 min read
World Models HandbookAll chapters
Part 11 of 12Evaluation and Understanding
Chapter 63State, Physics, Causality, and Memory Evaluation › Physical Rule and Constraint Tests
Reading section 4 of 8 · Physical Rule and Constraint Tests · 11 minSection 4 of 8 · 11 min

Book contents

Section 4 of 8 · 11 min read

State, Physics, Causality, and Memory Evaluation › Physical Rule and Constraint TestsOne section · Sections in this chapter4 / 8 ↓
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The previous section asked whether the hidden state is coherent. This one asks whether the dynamics obey a specified physical law under a named regime. A model can produce a smooth, coherent, plausible trajectory that travels at an unphysical speed through a dissipative medium, or that passes through a wall, or that gains energy at every bounce. Constraint tests are how you catch that. They are the natural complement to the trajectory-accuracy tests above: accuracy asks whether the model is close to the truth on average, and constraint tests ask whether the model respects the mechanisms that produce the truth.

Specify the regime before testing a rule. In this force-free, elastic toy, kinetic energy is conserved. Applied forces can do work; damping and lossy contacts can dissipate energy, so a driven dissipative system needs both contributions in its energy balance. A conservation test without a regime can penalize correct dynamics. Here a lossy contact reduces normal kinetic energy only when the incident normal velocity is nonzero.

We will evaluate three regimes, all with the same dynamics code but different parameters:

Energy behaviour in the three physical regimes used for constraint tests.
RegimeForcingDampingRestitutionExpected energy behaviour over a rollout
Force-free, elasticnone1.01.0EH/E0=1E_H / E_0 = 1 exactly
Force-free, dampednone0.91.0EH/E0=0.92H=0.81HE_H / E_0 = 0.9^{2H} = 0.81^{H} exactly, since energy scales as v2v^2
Free, lossy contactnone1.00.7constant between contacts; normal-component energy is multiplied by 0.490.49 at an impact

The damping factor multiplies both velocity components by 0.90.9 each step, so total kinetic energy is multiplied by 0.810.81. At a wall, restitution multiplies only the normal velocity by −e-e; its energy component is multiplied by e2=0.49e^2=0.49, while tangential energy is unchanged. Total energy therefore does not generally fall by a factor of 0.490.49. These regimes test different mechanisms, but a residual alone cannot identify which mechanism is wrong.

The metric: end-of-rollout energy ratio residual

For a rollout of length H=30H = 30 starting from the true initial state s0s_0, define the predicted energy ratio ρ^=E^H/E^0\hat\rho = \hat{E}_H / \hat{E}_0 and the true ratio ρ=EH/E0\rho = E_H / E_0, where EE is the kinetic energy

Et=12m∥vt∥22E_t = \tfrac{1}{2} m \|\mathbf{v}_t\|_2^2

where m=1m = 1 kg is the mass and vt\mathbf{v}_t is the velocity at step tt. The energy ratio residual is

ERR=∣ρ^−ρ∣\mathrm{ERR} = \big| \hat\rho - \rho \big|

with the predicted and true ratios defined as

ρ^=E^HE^0,ρ=EHE0\hat\rho = \frac{\hat{E}_H}{\hat{E}_0}, \qquad \rho = \frac{E_H}{E_0}

where:

  • HH: the rollout horizon, 3030 steps, so HΔt=3.0H\Delta t = 3.0 s.
  • E^t\hat{E}_t: the kinetic energy of the model's predicted state at step tt.
  • EtE_t: the kinetic energy of the simulator's true state at step tt.
  • ρ^\hat\rho: the model's predicted energy ratio over the rollout.
  • ρ\rho: the true energy ratio produced by the simulator's own dynamics.

These ratios require positive initial energy. The fixtures satisfy that condition; the implementation floors denominators at 10−1210^{-12} joules as a numerical safeguard, which changes the ratio convention near zero. In the elastic case ρ=1\rho=1; with damping and no clipping ρ=0.81H\rho=0.81^H; with lossy contact the ratio depends on each impact. The violation rate uses the declared absolute ratio-error tolerance 0.050.05, not a universal physical threshold.

Why not just check Et+1=EtE_{t+1} = E_t?

Stepwise kinetic-energy equality is the relevant rule only in the force-free, elastic regime here. A correct simulation may have floating-point discrepancies, whereas a wrong predictor may have much larger errors. Comparing with the regime's reference ratio and stating a tolerance makes the evaluated quantity explicit.

We evaluate four predictors in each regime to compare regime dependence and model-class limitations. Simulator truth is reused as an identity reference; evaluating numerical integration drift would require an independent higher-accuracy reference or a time-step convergence study, which this example does not provide.

  • True integrator. Simulator truth reused as the reference prediction. Its zero residual follows by identity, not from an independent numerical-accuracy check.
  • Linear model fitted on force-free data. A one-step linear map s^t+1=Ast+b\hat{s}_{t+1} = A s_t + b estimated by ridge regression on force-free trajectories.
  • Linear model fitted on damped data. The same estimator, fitted on damped trajectories.
  • No-change control. The deliberately wrong negative control: s^t+1=st\hat{s}_{t+1} = s_t. It preserves initial velocity and kinetic energy but freezes position, illustrating that passing an energy check is not sufficient for correct dynamics.
In[22]:
Code
def kinetic_energy(vel, mass=1.0):
    return 0.5 * mass * np.sum(vel**2, axis=-1)


def fit_linear_dynamics(pos, vel):
    """Ridge fit of s_{t+1} ~ [s_t, 1] using all transitions in a regime."""
    S = np.concatenate([pos, vel], axis=-1)
    X = S[:, :-1].reshape(-1, S.shape[-1])
    Y = S[:, 1:].reshape(-1, S.shape[-1])
    return ridge_fit(np.hstack([X, np.ones((X.shape[0], 1))]), Y, alpha=1e-6)


def rollout(W, s0, n_steps):
    """Recursively apply the learned one-step model. s0 has shape (n, 4)."""
    S = np.empty((s0.shape[0], n_steps + 1, s0.shape[1]))
    S[:, 0] = s0
    for t in range(n_steps):
        Xb = np.hstack([S[:, t], np.ones((s0.shape[0], 1))])
        S[:, t + 1] = Xb @ W
    return S


def evaluate(S_pred, truth, tol=0.05):
    E_pred = kinetic_energy(S_pred[..., 2:4])
    E_true = kinetic_energy(truth[..., 2:4])
    ratio_pred = E_pred[:, -1] / np.maximum(E_pred[:, 0], 1e-12)
    ratio_true = E_true[:, -1] / np.maximum(E_true[:, 0], 1e-12)
    resid = float(np.mean(np.abs(ratio_pred - ratio_true)))
    viol = float(np.mean(np.abs(ratio_pred - ratio_true) > tol))
    pos_err = np.linalg.norm(S_pred[..., :2] - truth[..., :2], axis=-1)
    return resid, viol, np.sqrt(np.mean(pos_err**2, axis=0))
In[23]:
Code
H_TEST = 30
rng_e = np.random.default_rng(2024)

regimes = {
    "free / elastic": simulate(
        rng_e, 20, driven=False, damping=1.0, restitution=1.0
    ),
    "free / damped": simulate(
        rng_e, 20, driven=False, damping=0.9, restitution=1.0
    ),
    "free / lossy contact": simulate(
        rng_e, 20, driven=False, damping=1.0, restitution=0.7, v_scale=1.5
    ),
}
fit_sets = {
    "free": simulate(rng_e, 20, driven=False, damping=1.0, restitution=1.0),
    "damped": simulate(rng_e, 20, driven=False, damping=0.9, restitution=1.0),
}

W_free = fit_linear_dynamics(fit_sets["free"][0], fit_sets["free"][1])
W_damp = fit_linear_dynamics(fit_sets["damped"][0], fit_sets["damped"][1])

phys_results = {}
for rname, (pos_r, vel_r, act_r, obs_r) in regimes.items():
    s0 = np.concatenate([pos_r[:, 0], vel_r[:, 0]], axis=-1)
    truth = np.concatenate(
        [pos_r[:, : H_TEST + 1], vel_r[:, : H_TEST + 1]], axis=-1
    )
    models = {
        "true integrator": truth,
        "linear fit (free)": rollout(W_free, s0, H_TEST),
        "linear fit (damped)": rollout(W_damp, s0, H_TEST),
        "no-change control": np.repeat(s0[:, None, :], H_TEST + 1, axis=1),
    }
    phys_results[rname] = {m: evaluate(S, truth) for m, S in models.items()}
Out[24]:
Console
free / elastic
  model                   energy ratio resid  violation rate   RMSE h=1  RMSE h=30
  true integrator                     0.0000           0.000    0.00000    0.00000
  linear fit (free)                   0.9537           1.000    0.00100    0.16624
  linear fit (damped)                 0.9982           1.000    0.00254    0.49824
  no-change control                   0.0000           0.000    0.02536    0.71493

free / damped
  model                   energy ratio resid  violation rate   RMSE h=1  RMSE h=30
  true integrator                     0.0000           0.000    0.00000    0.00000
  linear fit (free)                   0.0728           0.500    0.00368    0.41512
  linear fit (damped)                 0.0000           0.000    0.00000    0.00000
  no-change control                   0.9982           1.000    0.02568    0.24587

free / lossy contact
  model                   energy ratio resid  violation rate   RMSE h=1  RMSE h=30
  true integrator                     0.0000           0.000    0.00000    0.00000
  linear fit (free)                   0.5171           1.000    0.00054    0.25837
  linear fit (damped)                 0.5730           1.000    0.00413    0.34631
  no-change control                   0.4252           0.850    0.04129    0.58632
Out[25]:
Visualization
Grouped bar chart of energy ratio residual across three regimes for four dynamics models.
End-of-rollout energy-ratio error relative to simulator truth across three regimes. The matched damped linear fit performs well on damped trajectories, but the free-trained global linear model fails to preserve elastic dynamics over this horizon. The no-change control passes the elastic energy test despite incorrect positions. Constraint fidelity and trajectory fidelity must be checked separately.

The simulator trajectory is compared with itself and therefore has exactly zero residual; this is a reference identity check, not an independent numerical-accuracy test. The damped fit matches damped data closely. The free-trained global linear map cannot represent the piecewise reflection mechanism and has a large elastic-regime energy error. The no-change control preserves energy in that regime while freezing position, so its zero energy error does not certify useful dynamics. The tolerance of 0.05 is an explicit diagnostic choice, not a universal physical threshold.

Out[26]:
Visualization
Line chart of recursive rollout position RMSE against horizon for four dynamics models.
Position RMSE over recursive rollouts in the elastic regime. Both global linear fits accumulate error, although the free-trained fit performs better at the final horizon. The no-change control illustrates poor trajectory fidelity despite passing the elastic energy test. Initial-state error at horizon zero is zero for every method by construction.

Separating the sources of a residual

A non-zero constraint residual can come from at least five places, and attributing it correctly is most of the interpretive work:

  • Learned dynamics error. The model misrepresents the mechanism.
  • Numerical integration drift. Under-resolved integration can introduce error. Diagnosing it needs an independent numerical reference; our identity comparison does not measure it.
  • Observation error. If the rollout is initialised from noisy observations rather than the true state, the initial energy estimate itself is noisy.
  • Coordinate and unit errors. Inconsistent velocity conversion across initial and final states can distort ratios. A common multiplicative conversion of all velocities cancels in EH/E0E_H/E_0; ratio agreement cannot detect every unit error.
  • A mismatched regime. The model may be perfectly correct for a different regime than the one under test.

The experiment includes regime mismatch and model-class misspecification. A systematic signed residual can arise from either, from calibration bias, or from other errors; its sign does not uniquely diagnose the cause. Compare matched-regime data, held-out mechanisms and an appropriate numerical reference before assigning responsibility.

The next figure compares the free-trained and damped-trained linear fits on the same 20 elastic held-out rollouts, plotting each unit's signed energy-ratio residual instead of its average. The fits also use different realized initial-state draws in their training trajectories, so this comparison does not isolate a change in training regime alone.

In[27]:
Code
## Signed energy ratio residuals on the same 20 elastic rollouts, one value per unit.
def signed_ratio_residual(S_pred, truth):
    E_pred = kinetic_energy(S_pred[..., 2:4])
    E_true = kinetic_energy(truth[..., 2:4])
    rho_pred = E_pred[:, -1] / np.maximum(E_pred[:, 0], 1e-12)
    rho_true = E_true[:, -1] / np.maximum(E_true[:, 0], 1e-12)
    return rho_pred - rho_true


pos_el, vel_el, _, _ = regimes["free / elastic"]
s0_el = np.concatenate([pos_el[:, 0], vel_el[:, 0]], axis=-1)
truth_el = np.concatenate(
    [pos_el[:, : H_TEST + 1], vel_el[:, : H_TEST + 1]], axis=-1
)

residual_matched = signed_ratio_residual(
    rollout(W_free, s0_el, H_TEST), truth_el
)
residual_mismatched = signed_ratio_residual(
    rollout(W_damp, s0_el, H_TEST), truth_el
)

## Unique trajectory IDs keep all 20 units countable even when values coincide.
trajectory_ids = np.arange(1, residual_matched.size + 1)
Out[28]:
Visualization
Signed energy ratio residuals for free-trained and damped-trained linear fits, with each of the twenty test trajectory IDs on its own row.
Signed energy-ratio error for 20 elastic test trajectories. Both global linear fits underestimate final relative energy, including the free-trained fit. The unit-level view reveals recurring error but does not identify regime mismatch as its unique cause.

Constraint projection can improve a chosen residual while worsening trajectory prediction. Renormalizing a nonzero predicted velocity to its initial magnitude enforces energy conservation in the elastic regime, but is wrong for a dissipative regime's energy target. Clamping position to the box does not enforce energy conservation. A projection must target the actual constraint and satisfy its feasibility conditions; it does not universally drive every residual to zero. Report projected and unprojected errors together.

Finally, note that constraint testing is not a substitute for the state probes of the previous section. A model can respect every conservation law while tracking the wrong object, and it can track the right object while violating energy conservation. These are independent axes of evaluation. The full picture requires both.