State, Physics, Causality, and Memory Evaluation › Object Permanence and Temporal Consistency
World Models HandbookAll chapters State, Physics, Causality, and Memory Evaluation › Object Permanence and Temporal ConsistencyOne section · Sections in this chapter3 / 8 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Our probe evaluates individual timesteps. Object permanence concerns an object's continued existence when it is unobserved. Tracking its hidden position through a gap is a richer operational diagnostic: a model may preserve existence or identity while remaining uncertain about location or predicting motion inaccurately. Hidden-trajectory evaluation therefore needs its own target in addition to a persistence criterion.
This is where compounding error and memory meet. The previous chapter, "Perceptual and Generative Evaluation," asked whether a generated frame looks right. Here we ask whether the hidden trajectory is right. For an occluded-object control task, position accuracy can matter to the chosen action even when the generated frames look plausible. These are complementary tests: accurate hidden positions do not guarantee good visual generation, and plausible frames do not certify accurate positions.
We should be precise about what "correct" means during an occlusion. Three distinct properties are usually bundled together:
- Hidden trajectory accuracy. The model's estimate of the ball's position while hidden is close to the true position.
- Reappearance accuracy. At the first visible timestep after the gap, the model's belief matches the observation.
- Identity persistence. The object that reappears is the same object, rather than a newly spawned one that happens to be nearby.
Multiple indistinguishable objects can exchange identities while preserving the visible set of positions. An identity-aware test needs persistent labels or another identity criterion. Our single-object toy does not test competing identity assignments; that does not make identity and trajectory accuracy equivalent. A tracked object can keep its label while its predicted position is wrong.
Observational ambiguity also matters. A noisy visible history may be consistent with several hidden continuations. A point loss can still compare predictors by expected risk over realized trajectories, but does not characterize the forecast distribution. Where the task needs uncertainty, report and evaluate a conditional distribution rather than treating one plausible continuation as the uniquely predictable outcome.
Setting up the occluded-object task
The six paths below are the first six held-out trajectories in index order, and the amber markers identify frames hidden by the band. This figure shows ground-truth paths and visibility geometry, not a predictor's errors or reappearance accuracy.
## Deterministic selection rule: the first six held-out trajectories in index order,
## with no resampling and no random draw. SHOW_IDX is fixed at construction time.
N_SHOW = 6
SHOW_IDX = np.arange(N_SHOW)
show_pos = pos_all[test_sl][
SHOW_IDX
] # (N_SHOW, N_STEPS + 1, 2) positions in metres
show_mask = mask_all[test_sl][SHOW_IDX] # 1.0 visible, 0.0 behind the occluder
occ_lo, occ_hi = 0.5 - OCC_HALF, 0.5 + OCC_HALF
We evaluate three predictors with explicitly stated information budgets. Stating the budget matters here more than in almost any other comparison, because the three predictors differ enormously in what they are allowed to see, and a naive comparison across them is meaningless.
- Freeze. Holds the last visible observation fixed: . Uses observations only, no motion model. On a free trajectory without a wall bounce during the gap, this drifts at the object's speed, so its error grows roughly linearly in and is the natural last-observation, no-motion baseline.
- Constant velocity. Estimates velocity from the two immediately preceding visible observations and extrapolates: . It ignores subsequent forces and contacts. Observation noise and those omitted dynamics can all affect its error.
- Privileged constant velocity. Extrapolates the simulator's true position and velocity at , but still ignores subsequent forces and contacts. This baseline discloses simulator-state access; it is neither a full-simulator oracle nor an irreducible-error bound.
The metric is the mean Euclidean position error in metres over the hidden window, indexed by the hidden offset . Formally, writing for the set of occluded runs that survive to offset and for a predictor's estimate on run at hidden step , the reported quantity at offset is
where:
- : the hidden offset in steps of s, running from to .
- : an index over occluded runs in the held-out split.
- : the index of the last visible timestep before run , so is a hidden timestep.
- : the predictor's estimated position at that hidden step.
- : the simulator's true position at that hidden step.
- : the denominator, the number of occluded runs that are at least steps long.
The metric is in metres and is an average over runs at each offset, with the same occluded-run denominator applied to every predictor. The denominator shrinks as grows, since fewer runs survive to large offsets, and we report it alongside the error so you can judge how thin the tail is. Every predictor is evaluated on the same , while their different information budgets remain explicit. At a fixed offset, each eligible run contributes once; this is not a pooled average over all hidden frames.
import numpy as np
## Wall bounces during occlusion are already produced by the reflecting-wall
## dynamics in simulate(); this evaluation reuses the main held-out trajectories
## used by the probes and freeze-drift diagnostic. The bounces come from the physics,
## not from a post-hoc perturbation, so no trajectory is modified after the fact.
def occluded_runs(vis_row):
"""Return maximal (start, stop) index intervals where visibility is 0."""
runs, start = [], None
for i, value in enumerate(vis_row):
if value < 0.5 and start is None:
start = i
elif value >= 0.5 and start is not None:
runs.append((start, i))
start = None
if start is not None:
runs.append((start, len(vis_row)))
return runs
def first_usable_run(vis_row, min_start=2):
"""First hidden run with two immediately preceding visible frames."""
for start, stop in occluded_runs(vis_row):
if (
start >= min_start
and stop > start
and np.all(vis_row[start - 2 : start] >= 0.5)
):
return start, stop
return None
## Regression fixtures: earlier visibility somewhere in the history is not enough.
assert first_usable_run(np.array([1, 0, 1, 0, 0, 1])) is None
assert first_usable_run(np.array([1, 1, 0, 0, 1])) == (2, 4)
assert first_usable_run(np.array([0, 0, 0, 0])) is None
assert first_usable_run(np.array([1, 1, 1, 1])) is None
assert occluded_runs(np.array([1, 1, 0, 0])) == [(2, 4)]K_MAX = 6
pos_te = pos_all[test_sl]
obs_te = obs_all[test_sl]
vel_te = vel_all[test_sl]
mask_te = mask_all[test_sl]
predictor_names = [
"freeze",
"constant velocity",
"privileged constant velocity",
]
err_sum = {name: np.zeros(K_MAX) for name in predictor_names}
n_at_k = np.zeros(K_MAX)
for i in range(pos_te.shape[0]):
for start, stop in occluded_runs(mask_te[i]):
if start < 2 or not np.all(mask_te[i, start - 2 : start] >= 0.5):
continue
p_last = obs_te[i, start - 1]
v_hat = (obs_te[i, start - 1] - obs_te[i, start - 2]) / DT
s_true_last = pos_te[i, start - 1]
v_true = vel_te[i, start - 1]
for k in range(1, min(stop - start, K_MAX) + 1):
j = start - 1 + k
if j >= pos_te.shape[1]:
break
truth = pos_te[i, j]
preds = {
"freeze": p_last,
"constant velocity": p_last + k * DT * v_hat,
"privileged constant velocity": s_true_last + k * DT * v_true,
}
for name, pred in preds.items():
err_sum[name][k - 1] += np.linalg.norm(pred - truth)
n_at_k[k - 1] += 1
mean_err = {
name: np.divide(
err_sum[name], n_at_k, out=np.full(K_MAX, np.nan), where=n_at_k > 0
)
for name in predictor_names
}
## An empty contributing set must be missing, not a fabricated zero error.
_empty_error = np.divide(
np.zeros(2), np.zeros(2), out=np.full(2, np.nan), where=np.zeros(2) > 0
)
assert np.isnan(_empty_error).all() hidden k freeze const. vel priv.CV n runs
1 0.0283 0.0100 0.0024 23
2 0.0539 0.0165 0.0050 22
3 0.0801 0.0242 0.0083 22
4 0.1066 0.0323 0.0121 22
5 0.1350 0.0412 0.0172 19
6 0.1573 0.0482 0.0223 18
Errors are mean Euclidean position error in metres, over held-out runs.
Privileged CV reads the last visible simulator state, not future forces.The table format is deliberate: the same metric, the same denominator, and the same test trajectories for all three predictors, with the information budget of each written down explicitly. This is what makes the comparison readable. Without the disclosed budgets, a reader would have no way to know whether the difference between two rows reflects a modelling improvement or a difference in what the predictors were allowed to see.

The shape of each curve is the diagnosis. The freeze predictor grows approximately linearly in here because it ignores the ball's velocity. Its error is the displacement from the last noisy observation to the realized hidden position, not the path length traveled. Under approximately straight, constant motion and small observation noise, the leading-order relation is
where:
- : the hidden offset in steps.
- : the ball's velocity vector.
- : the mean speed.
- s: the step size.
That is, the hidden offset times the mean speed times the step size, under approximately constant motion. The constant-velocity estimate improves on freezing here, but its residual can reflect observation noise, later driving actions, and contact changes. The privileged baseline extrapolates the true initial position and velocity rather than executing the full simulator. Its non-zero residual cannot be attributed uniquely to wall bounces, and it is not an irreducible-error floor.
The dashed line uses , with mean speed over all hidden held-out frames. That population differs from the eligible run set at each offset, and driving, contacts, and measurement noise also affect the measured curve. Their similarity here is a descriptive comparison, not an exact matched-population identity or a causal decomposition.
## Leading-order prediction of the freeze error: k * dt * mean speed on hidden frames.
## The mean speed is a deterministic statistic of vel_te and mask_te, with no sampling,
## using the same main held-out trajectories as the hidden-horizon predictors.
hidden_frames = mask_te < 0.5
mean_speed_hidden = float(np.linalg.norm(vel_te[hidden_frames], axis=-1).mean())
drift_pred = np.arange(1, K_MAX + 1) * DT * mean_speed_hidden
freeze_measured = mean_err["freeze"]
The comparison names each predictor's information budget and metric denominator. In this driven task, horizon-dependent error mixes initial-state estimation with omitted future forces and possible contacts. It is not a pure measure of learned memory: these predictors are handcrafted, and the constant-velocity baseline has no learned recurrent state.
Sweeping the occlusion length
Horizon is one axis; occluder geometry is another. We sweep the band half-width on fixed undriven trajectories. The metric is position error at the final hidden step of the first usable gap, not at the first visible reappearance. Wider bands change both gap lengths and which trajectories qualify, so this sweep does not isolate elapsed time from sample selection.
sweep_widths = [0.02, 0.05, 0.10, 0.20, 0.30]
rng_sw = np.random.default_rng(404)
p_sw, v_sw, a_sw, o_sw = simulate(rng_sw, 40, driven=False)
sweep_rows = []
for w in sweep_widths:
m_w = visibility(p_sw, occ_half=w)
f_tot = c_tot = t_tot = 0.0
n_used = 0
n_censored = 0
gap_lengths = []
for i in range(p_sw.shape[0]):
run = first_usable_run(m_w[i])
if run is None:
continue
start, stop = run
k = stop - start
if k < 1:
continue
truth = p_sw[i, start - 1 + k]
p_last = o_sw[i, start - 1]
v_hat = (o_sw[i, start - 1] - o_sw[i, start - 2]) / DT
f_tot += np.linalg.norm(p_last - truth)
c_tot += np.linalg.norm(p_last + k * DT * v_hat - truth)
t_tot += np.linalg.norm(
p_sw[i, start - 1] + k * DT * v_sw[i, start - 1] - truth
)
n_used += 1
n_censored += int(stop == m_w.shape[1])
gap_lengths.append(k)
denominator = n_used if n_used else np.nan
sweep_rows.append(
(
w,
float(np.mean(gap_lengths)) if n_used else np.nan,
f_tot / denominator,
c_tot / denominator,
t_tot / denominator,
n_used,
n_censored,
)
) occ. half-width mean gap freeze const. vel priv.CV n censored
0.02 1.70 0.0405 0.0177 0.0000 40 0
0.05 4.38 0.1057 0.0412 0.0000 40 0
0.10 8.68 0.2105 0.0658 0.0000 40 1
0.20 17.11 0.4097 0.0985 0.0000 36 8
0.30 21.57 0.4776 0.1263 0.0000 21 12
Error is measured at the last hidden step of the first gap in each rollout.
All three predictors are evaluated on the same trajectories and the same gaps.The freeze predictor accumulates larger errors as the band widens; the observation-based constant-velocity predictor remains better but also degrades. The privileged extrapolator is essentially exact on these evaluated gaps. This does not demonstrate a bounce-induced jump. A gap still hidden at the simulation endpoint is right-censored: its recorded length is only the observed portion. Report contributing trajectories and censored counts alongside that mean length.

Memory ablations, and what they do and do not show
Recurrent and transformer-based world models carry their hidden state across time in a learned memory. A natural diagnostic is to reset or truncate that memory at the moment the object disappears and see whether performance degrades. If it does, the memory matters for the task. This is an appealing experiment because it is easy to run, but the inference it supports is narrower than it first appears.
A mid-episode reset may be out of distribution if training never used such resets. Context truncation may likewise be unfamiliar, depending on the training protocol. Performance changes measure sensitivity to the stated manipulation, not an isolated memory circuit. Describe the training context and reset schedule before interpreting either ablation.
Four useful temporal properties need different measurements:
- Sequence coherence is a local property: successive predicted states are mutually consistent.
- Long-range dependence is about how far back in time the model's prediction depends on.
- Object identity requires a persistent identity criterion; multi-object gaps make assignment ambiguity particularly visible.
- Uncertainty should be reported, not suppressed: when several hidden continuations are consistent with the visible data, a good model produces a spread of plausible futures rather than one confident guess.
Finally, note the discipline about oracle baselines and information budgets. It is easy to build a comparison in which one method quietly reads the simulator's latent state while another reads only observations, and then report the resulting gap as a modelling improvement. Every baseline in this section is labelled with what it reads. If you introduce an oracle, introduce it loudly. The reason this discipline is worth repeating is that the temptation to compare against an oracle without disclosing it is strong: the oracle often performs better, and the difference can be presented as evidence of modelling quality rather than of information asymmetry.