The World-Model Qualification Test › A Worked Experiment: Two Models, One Plant
World Models HandbookAll chapters The World-Model Qualification Test › A Worked Experiment: Two Models, One PlantOne section · Sections in this chapter7 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
We will build a deliberately simple physical system so that every number is checkable by hand: a damped cart moving in one dimension under a force input, observed through a noisy position sensor. The true state is the pair (position, velocity), but the sensor reports only position. This single design choice creates the entire drama: a model that looks only at the latest observation is missing velocity, which is exactly the quantity that determines where the cart goes next.
Simplicity here is a feature, not a concession. Because the true dynamics are known analytically, we can compute an oracle bound on performance, verify every diagnostic by hand, and attribute every observed difference between the two models to exactly one cause: the presence or absence of history in the input. In a realistic robotics experiment, dozens of confounds (network capacity, optimization noise, data volume, reward shaping) move simultaneously, and clean attribution becomes impossible. The pedagogical value of a system you can fully understand is that it lets you see the criteria working before you apply them to systems you cannot.
The environment's transition is a discretized double integrator with damping:
where:
- : the cart's velocity at time
- : the cart's true position at time
- : the acceleration command the agent chooses at time
- : a random disturbance force that keeps the control problem honest (without it, doing nothing would be optimal)
- : the damping coefficient
- : the timestep
The agent observes , where is sensor noise.
Reading the equations physically: the first line says velocity updates under the commanded acceleration, a random disturbance, and a damping term proportional to current velocity, so the cart naturally slows unless pushed. The second line says position advances by the new velocity, a semi-implicit discretization choice that keeps the simulation stable at modest timesteps. Together they form a double integrator, the canonical minimal model of mechanical motion, dressed with just enough realism (damping, disturbance, noise) that none of our conclusions will feel like artifacts of a frictionless textbook fantasy.
We will train two models on identical data:
- Model M (memoryless): predicts the next position from the current position and action alone, , where is the fitted weight vector.
- Model H (history-based): predicts from the last three positions and the action, , where is the fitted weight vector.
Both are linear models fit by ridge regression, so the comparison changes one factor: how much observation history is available. Model M cannot explicitly reconstruct velocity. Model H has enough inputs to form a noisy velocity estimate, but whether three observations are sufficient is a result to test, not an assumption.
Why three positions specifically? Two consecutive noisy positions give a crude finite-difference velocity estimate, but the noise in each observation contaminates the difference heavily. Three positions begin to average that contamination down, giving the ridge fit enough signal to recover a usable implicit velocity feature while keeping the feature vector small enough to inspect by eye. The choice also foreshadows a general principle: the window length needed for sufficiency scales with how quickly the hidden state can be inferred from observations relative to the observation noise, a trade-off that filtering theory makes precise and that we revisit in Part II.
Building the environment and collecting data
All modeling and numerical computation in this chapter uses NumPy; Matplotlib renders the figures. First, the environment and a data-collection routine.
from collections import deque
import numpy as np
DT, DAMPING = 0.1, 0.6 # timestep and damping coefficient
OBS_NOISE = 0.03 # sensor noise standard deviation
DIST_STD = 0.5 # disturbance force standard deviation
def env_step(p, v, action, rng):
"""Advance the true plant one step; returns next position and velocity."""
disturbance = rng.normal(0.0, DIST_STD)
v_next = v + DT * (action + disturbance - DAMPING * v)
p_next = p + DT * v_next
return p_next, v_next
def collect_trajectory(policy, n_steps, rng):
"""Roll out the plant under a policy; returns (observations, actions)."""
p, v = 0.0, 0.0
obs = [p + rng.normal(0.0, OBS_NOISE)]
acts = []
for _ in range(n_steps):
a = policy(p, rng)
acts.append(a)
p, v = env_step(p, v, a, rng)
obs.append(p + rng.normal(0.0, OBS_NOISE))
return np.array(obs), np.array(acts)
random_policy = lambda p, rng: rng.uniform(-1.0, 1.0)
rng = np.random.default_rng(7)
train_trajs = [collect_trajectory(random_policy, 250, rng) for _ in range(40)]
test_trajs = [collect_trajectory(random_policy, 250, rng) for _ in range(20)]Train trajectories: 40 x 250 transitions Test trajectories: 20 x 250 transitions
The training data comes from uniformly random actions. This choice matters: random exploration spreads probability mass across the action space, which gives both models their best possible shot at action conditioning. Later, in the counterfactual experiment, we will see what happens when this generosity is withdrawn.
Note also what the random policy buys us beyond coverage: because actions are independent of states, the observational and interventional distributions coincide in this dataset, satisfying the exogeneity condition from Criterion 2 by construction. Any action-conditioning failure we observe later can therefore be attributed to the model rather than to confounding in the data. This is the experimental-design analogue of a randomized trial, and it is worth remembering that such cleanliness is a luxury purchased with exploration budget, one that real systems rarely enjoy.
Fitting the two candidate models
Next we assemble regression datasets from the trajectories and fit both models by ridge regression. Each training example is a window of past observations plus the action taken, with the next observation as the target.
def fit_ridge(X, y, lam=1e-4):
"""Closed-form ridge regression: w = (X^T X + lam I)^{-1} X^T y."""
d = X.shape[1]
return np.linalg.solve(X.T @ X + lam * np.eye(d), X.T @ y)
def build_dataset(trajs, feat_fn, window):
"""Stack (window of observations, action) -> next observation pairs."""
X, y = [], []
for obs, acts in trajs:
for t in range(window - 1, len(acts)):
X.append(feat_fn(obs[t - window + 1 : t + 1], acts[t]))
y.append(obs[t + 1])
return np.array(X), np.array(y)
mem_feats = lambda w, a: [1.0, w[-1], a] # [bias, o_t, a_t]
hist_feats = lambda w, a: [1.0, w[-1], w[-2], w[-3], a] # adds o_{t-1}, o_{t-2}
X_mem, y_mem = build_dataset(train_trajs, mem_feats, window=1)
X_hist, y_hist = build_dataset(train_trajs, hist_feats, window=3)
w_mem = fit_ridge(X_mem, y_mem)
w_hist = fit_ridge(X_hist, y_hist)Memoryless weights [bias, o_t, a_t]: [2.0000e-04 1.0042e+00 1.1300e-02]
History weights [bias, o_t, o_{t-1}, o_{t-2}, a_t]: [ 2.000e-04 8.466e-01 3.146e-01 -1.569e-01 1.100e-02]Before looking at any score, read the weights: they encode what each model believes about physics. The history model's coefficients on consecutive positions combine into an implicit velocity estimate, so its action coefficient should approach the true one-step displacement per unit action, roughly . The memoryless model cannot represent velocity at all, so its action coefficient absorbs whatever the data's correlations allow and no more.
This weight-inspection habit generalizes far beyond linear models. Whenever your model class admits human-readable parameters, reading them before reading scores converts an opaque fitting exercise into a hypothesis test: you wrote down what correct physics would imply, and now you check whether the fit agrees. With neural networks the parameters stop being readable, but the practice survives in modified form as probing, which is exactly the diagnostic Criterion 1 recommended. The underlying discipline is the same: interrogate the representation directly instead of inferring its contents solely from end-task behavior.
Criterion 1 applied: one-step error and the sufficiency diagnostic
We evaluate one-step prediction error on held-out random-action trajectories, then run the incremental-history diagnostic: does giving the memoryless model more history fix it?
def one_step_mse(w, feat_fn, trajs, window):
X, y = build_dataset(trajs, feat_fn, window)
resid = y - X @ w
return float(np.mean(resid**2))
mse_mem = one_step_mse(w_mem, mem_feats, test_trajs, window=1)
mse_hist = one_step_mse(w_hist, hist_feats, test_trajs, window=3)One-step MSE, memoryless model: 0.00223 One-step MSE, history model: 0.00208 Ratio (memoryless / history): 1.07x Noise lower-bound reference: 0.00093
The result is deliberately less tidy than the hypothesis. Three observations reduce one-step MSE from about to , only a 7% improvement. Both errors remain more than twice the simple irreducible reference below. History contains useful information, but this comparison does not establish that a three-observation state is sufficient or that all remaining error is noise.
The reference line combines the next observation's sensor-noise variance with the fresh disturbance's direct one-step contribution to position:
It is a lower-bound reference, not the exact attainable error for these finite noisy histories. Uncertainty about velocity and estimation error can keep a fitted predictor above it. Reporting this reference is still useful, provided we do not relabel every gap above it as representation failure.
The history sweep now asks the sharper question: does the error flatten once the window is adequate, or does additional history continue to help?
def make_window_feats(n):
"""Feature function using the last n observations plus the action."""
return lambda w, a: [1.0] + list(w[-n:]) + [a]
history_windows = np.arange(1, 7)
history_sweep = []
for n in history_windows:
feats_n = make_window_feats(n)
w_n = fit_ridge(*build_dataset(train_trajs, feats_n, window=n))
history_sweep.append(one_step_mse(w_n, feats_n, test_trajs, window=n))
Read @fig-history-sweep together with the printed numbers above. Error falls by about 20% from a one-observation window to a six-observation window, and it continues falling at the right edge. The test therefore rejects the convenient claim that three observations are already sufficient. It does show that the one-observation representation discards predictive information. The next experiment asks whether that modest one-step difference grows over long recursive rollouts.
Criterion 3 applied: rollout error compounds differently
One-step scores flatter both models. Now we let each model predict autoregressively: feed back its own prediction as the next input, for fifty steps, against held-out ground truth. This is the open-loop version of what a planner experiences.
HORIZON = 50
def rollout_mse(predict, trajs, window, horizon):
"""Mean squared error per step of recursive multi-step prediction."""
errs, count = np.zeros(horizon), 0
for obs, acts in trajs:
for t in range(len(acts) - horizon):
buf = deque(obs[t : t + window], maxlen=window)
for h in range(horizon):
pred = predict(list(buf), acts[t + h])
errs[h] += (pred - obs[t + h + 1]) ** 2
buf.append(pred)
count += 1
return errs / count
predict_mem = lambda buf, a: float(np.array([1.0, buf[-1], a]) @ w_mem)
predict_hist = lambda buf, a: float(
np.array([1.0, buf[-1], buf[-2], buf[-3], a]) @ w_hist
)
err_mem = rollout_mse(predict_mem, test_trajs, window=1, horizon=HORIZON)
err_hist = rollout_mse(predict_hist, test_trajs, window=3, horizon=HORIZON)Two implementation details carry conceptual weight. First, the ground-truth actions are fed throughout the rollout, so this experiment isolates state-compounding from action-distribution shift; the planner-facing version of the same phenomenon, where the model must also digest its own planned actions, appears in the closed-loop test below. Second, the buffer is filled with predictions after the first step, which is precisely the feedback loop that distinguishes multi-step evaluation from repeated one-step evaluation. Every error the model makes becomes part of its own future input, and the experiment measures how gracefully each model tolerates dining on its own output.
Read the vertical axis: it is logarithmic. Both models deteriorate sharply, from errors near at one step to between and at step 50. The history model retains an advantage throughout, but the absolute gap does not explode; it becomes small relative to the shared long-horizon error. The defensible conclusion is not that one representation compounds while the other remains stable. It is that one-step validation understated the failure of both models at planning-length horizons.
This is also a useful negative result. Extra history improves the starting point, yet recursive prediction eventually overwhelms that advantage. State sufficiency and rollout stability are different properties: a better state can reduce error without making the learned transition stable under repeated self-input.
Criteria 2 and 4 applied: closing the loop
Now the decisive test. We place each model inside a model-predictive controller that plans by random shooting: sample many candidate action sequences, simulate each through the model, keep the sequence with lowest predicted cost, execute only its first action, observe, and replan. The task is regulation, holding the cart near the origin under random disturbance forces, with cost per step.
Random shooting is the simplest possible planner, and its simplicity makes the experiment interpretable. It can still exploit model error, but it searches less aggressively than a gradient optimizer. Replanning provides error correction: executing one action and consulting fresh observations limits how long imagination errors persist. The experiment will tell us whether the representation difference matters after that feedback correction.
We compare four agents: a blind policy that always outputs zero force, the memoryless model driving the planner, the history model driving the planner, and an oracle planner that cheats by simulating the true plant (including the hidden velocity) during its search. The oracle marks the ceiling achievable with perfect state knowledge; the gap between it and the learned models is the price of partial observability.
The baseline ladder matters as much as the headline numbers. The blind policy establishes what doing nothing achieves, so any claimed benefit of modeling must clear that bar. The oracle establishes what perfect information would achieve, so the distance from oracle to history model quantifies the remaining headroom attributable to estimation error rather than to fundamental difficulty. Without both anchors, a cost figure in isolation supports no conclusion: a mediocre result might reflect a hard task or a weak model, and only the anchors distinguish the two.
PLAN_HORIZON, N_CANDIDATES = 12, 128
EPISODE_STEPS, N_EPISODES = 120, 8
def plan(predict, buffer, rng):
"""Random-shooting MPC: return the first action of the best imagined sequence."""
candidates = rng.uniform(-1.0, 1.0, size=(N_CANDIDATES, PLAN_HORIZON))
best_cost, best_u = np.inf, 0.0
for seq in candidates:
buf, cost = list(buffer), 0.0
for u in seq:
p_next = predict(buf, u)
cost += p_next**2 + 0.05 * u**2
buf.append(p_next)
if cost < best_cost:
best_cost, best_u = cost, seq[0]
return best_u
def run_closed_loop(policy_fn, window, rng):
"""Run one episode; the agent observes noisy position only."""
p, v, cost = 0.0, 0.0, 0.0
buf = deque(np.zeros(window), maxlen=window)
for _ in range(EPISODE_STEPS):
u = policy_fn(list(buf), rng)
p, v = env_step(p, v, u, rng)
cost += p**2 + 0.05 * u**2
buf.append(p + rng.normal(0.0, OBS_NOISE))
return cost
def run_closed_loop_oracle(rng):
"""Same task, but the planner searches with the true (p, v) dynamics."""
p, v, cost = 0.0, 0.0, 0.0
for _ in range(EPISODE_STEPS):
best_cost, best_u = np.inf, 0.0
for seq in rng.uniform(-1, 1, size=(N_CANDIDATES, PLAN_HORIZON)):
pp, vv, c = p, v, 0.0
for u in seq:
vv = vv + DT * (u - DAMPING * vv)
pp = pp + DT * vv
c += pp**2 + 0.05 * u**2
if c < best_cost:
best_cost, best_u = c, seq[0]
p, v = env_step(p, v, best_u, rng)
cost += p**2 + 0.05 * best_u**2
return cost
blind = lambda buf, rng: 0.0
mpc_mem = lambda buf, rng: plan(predict_mem, buf, rng)
mpc_hist = lambda buf, rng: plan(predict_hist, buf, rng)
results = {}
for name, pol, win in [
("Blind (u=0)", blind, 1),
("Memoryless MPC", mpc_mem, 1),
("History MPC", mpc_hist, 3),
]:
costs = [run_closed_loop(pol, win, rng) for _ in range(N_EPISODES)]
results[name] = costs
oracle_costs = [run_closed_loop_oracle(rng) for _ in range(N_EPISODES)]
results["Oracle MPC"] = oracle_costs
names = list(results.keys())
means = [np.mean(results[n]) for n in names]
stds = [np.std(results[n]) for n in names] Blind (u=0): cost 48.29 (+/- 27.66)
Memoryless MPC: cost 2.75 (+/- 1.23)
History MPC: cost 2.46 (+/- 0.95)
Oracle MPC: cost 1.33 (+/- 0.14)The blind policy incurs a mean cost near . Both learned planners reduce that cost dramatically: about for the memoryless model and for the history model, compared with for the oracle. History helps, but not by the dramatic margin that the representation story alone might suggest. With eight stochastic episodes, the learned controllers' standard deviations overlap, so this run supports a modest advantage rather than a decisive ranking.
Why can the memoryless controller work at all? Model-predictive control executes only one imagined action before receiving a fresh observation and replanning. That feedback loop compensates for a weak latent state on this stable regulation task. The oracle gap shows that state estimation still leaves headroom, but the blind baseline prevents us from calling the memoryless controller a failure. Criterion 4 has done its job precisely because it overturned the tempting inference from representation diagnostics to task failure.
Notice what this experiment demonstrates about the relationship between the criteria. The offline tests found a representation weakness, but closed-loop replanning largely compensated for it on this task. Had we run only the one-step diagnostics, we might have predicted a much larger control deficit. Had we run only the closed-loop comparison, we would miss the reason the oracle still has an advantage and the possibility that a less forgiving task would expose it. The battery earns its length by revealing both the deficiency and the task's tolerance for it.
Stress-testing counterfactual validity
Finally, we withdraw the generous training data. Suppose the agent's predecessor operated a cautious feedback policy, with small Gaussian noise , so the logged data contains only gentle corrections around the origin. We retrain both models on this narrow diet and then interrogate them with two test sets: held-out transitions from the same cautious policy, and transitions under the wide random actions we used before.
This scenario mirrors the most common real-world data situation. Deployed systems log their own behavior, and deployed behavior is produced by controllers, which are smooth functions of state. The result is a dataset in which actions are nearly deterministic responses to observations, spanning only a thin manifold of the state-action space. Everything the qualification framework warns about concentrates in this setting: actions are confounded with states (breaking exogeneity), the action-space coverage collapses (limiting counterfactual validity), and the resulting models can look superb on held-out logs while being worthless for planning. The experiment below quantifies exactly how large the gap can be.
narrow_policy = lambda p, rng: np.clip(-2.0 * p + rng.normal(0, 0.1), -1, 1)
narrow_train = [collect_trajectory(narrow_policy, 250, rng) for _ in range(40)]
narrow_test = [collect_trajectory(narrow_policy, 250, rng) for _ in range(20)]
w_mem_n = fit_ridge(*build_dataset(narrow_train, mem_feats, window=1))
w_hist_n = fit_ridge(*build_dataset(narrow_train, hist_feats, window=3))
labels = ["Memoryless", "History"]
e_in = [
one_step_mse(w_mem_n, mem_feats, narrow_test, window=1),
one_step_mse(w_hist_n, hist_feats, narrow_test, window=3),
]
e_off = [
one_step_mse(w_mem_n, mem_feats, test_trajs, window=1),
one_step_mse(w_hist_n, hist_feats, test_trajs, window=3),
]Memoryless: in-distribution MSE 0.00174 | off-policy MSE 0.20443 | inflation 117.3x History: in-distribution MSE 0.00163 | off-policy MSE 0.10513 | inflation 64.6x
Both models degrade off-policy even though the physics never changed: error inflates by about for the memoryless model and for the history model. The models learned a useful local predictor under the cautious policy, not a globally valid account of the cart. Qualification is a joint property of model and data; extra history does not supply information about actions the policy never explored.
Nothing about the history model's architecture changed between the wide-data and narrow-data fits; only the behavior policy changed. Missing evidence about unexplored actions is not available for a flexible architecture to recover. Extrapolation then depends on inductive bias rather than identification. Data-collection strategy therefore belongs in the model-qualification record, not in a separate data-engineering footnote.
The action-margin probe makes the same point from a different angle. We hold the observation history fixed and sweep the action input across its full range. For this plant, the known one-step action effect is positive with gain . The question is therefore whether the learned line has the correct sign and scale.
action_grid = np.linspace(-1.0, 1.0, 41)
fixed_buffer = [0.0, 0.0, 0.0]
margin_wide = np.array([predict_hist(fixed_buffer, a) for a in action_grid])
margin_narrow = np.array(
[
float(
np.array(
[1.0, fixed_buffer[-1], fixed_buffer[-2], fixed_buffer[-3], a]
)
@ w_hist_n
)
for a in action_grid
]
)
@fig-action-margin-probe shows the interventional test in its simplest form. The wide-data coefficient is about , close to the expected . The narrow-policy coefficient is about : it has the wrong sign and roughly seven times the magnitude. Aggregate error shows that the model is fragile; this probe identifies the broken mechanism. The narrow fit has mistaken a policy correlation, stronger corrective actions occurring in states already moving toward recovery, for the physical effect of the action itself.