State, Physics, Causality, and Memory Evaluation › Limitations and Impact

Probing, occlusion sweeps, constraint tests, and intervention designs turn broad claims into narrower questions: which targets a readout recovers, how a predictor behaves while observations are hidden, which regime-specific constraints it respects, and which action effect a design identifies. The examples illustrate these procedures; they do not establish that the same measurements transfer unchanged to a trained world model.

The limitations are equally important, and they cluster into three themes.

The first three families use the reflecting point mass under named regimes; the causal example uses a separate one-step structural model. Their mechanisms, noise scales, and sampling policies are specified by construction, so the numerical values are illustrative rather than benchmarking evidence. Different fixtures can change the probe scores and residuals. The examples make the procedures explicit; they do not establish performance on a larger system.

Each diagnostic answers a specified question. A probe report measures recovery by a named procedure on a named split, not unrestricted knowledge or downstream use. The occluded-object tables compare three handcrafted predictors, not learned recurrent memory. The energy table checks global linear fits on both matched and mismatched regimes, alongside an identity reference and frozen-state control; it does not predict how a neural dynamics model would perform. Interpret each number within that scope.

The paired example cancels the shared disturbance exactly in its structural outcomes, while its noisy sample average estimates the slope approximately. Identification relies on the stated structure and design. Simulator results do not validate that structure for a real system; real interventions can, however, provide evidence that distinguishes some candidate mechanisms. Separate assumptions that a design checks from those it leaves untested.

Constraint satisfaction alone is insufficient. A projection enforcing the correct energy target can reduce that residual while worsening positions, and the frozen-state control passes the elastic energy test despite large position error. Our reused simulator truth is an identity reference, not an integration-accuracy floor.

Choose diagnostics for the task rather than treating the families as mandatory rungs. Probe accessibility with declared capacities, controls, and splits. Evaluate hidden windows with contributing-run and censoring counts. Test constraints under named regimes, alongside trajectory error and suitable reference baselines. For a causal target, state the identification design, overlap, feasibility, and structural assumptions before the estimate.

These offline diagnostics do not by themselves establish whether a planner using the model will make better decisions, rank policies correctly, or exploit an inaccurate region of the model. Chapter 64 evaluates executed returns alongside planner compute, policy ranking, and transfer; those decision diagnostics need not demonstrate a failure in every experiment. Benchmark taxonomy and statistical power are the subject of Chapter 65, and the mechanistic question of which internal components carry the state is taken up in Chapter 66. Failure modes and model exploitation are the opening subject of Chapter 67. Exploitation can make model-predicted returns optimistic, but a model's prediction gap can also be pessimistic.