State, Physics, Causality, and Memory Evaluation › State, Physics, Causality, and Memory Evaluation

Picture a small ball rolling across a table. For two seconds it is plainly visible. Then it passes behind a box, and for eight frames you see nothing at all. Then it emerges on the other side. Now ask a world model what happened in between.

This is a deceptively simple question. It sounds like a single question, but it is really a bundle of at least four distinct questions, and world models can answer some of them correctly while failing the others. That separation is the entire subject of this chapter.

There are at least four different things a model could do here, and they are easy to confuse. Among them, only the last is a causal question: the first three are observational or representational.

• It could generate eight frames that look like a ball continuing across the box. The pixels are plausible; the ball is in roughly the right place.
• It could maintain a hidden state that keeps the ball's position and velocity coherent, so that the ball emerges at exactly the right place at the right time.
• It could keep the ball's identity intact, so that the thing that emerges is the same object rather than a new one that happens to be nearby.
• It could do all of the above while also knowing that if you had pushed the ball differently just before it disappeared, it would have emerged somewhere else.

These are different requirements, not a strict hierarchy. Accurate positions do not guarantee persistent identity, and identity tracking does not guarantee accurate motion. An intervention asks a further question about changes to the world that observational prediction alone does not identify without assumptions. Each requirement needs its own evaluation protocol.

A model can pass the visual test while failing hidden-state, identity, or intervention tests. Even strong results across these offline diagnostics do not establish closed-loop usefulness: a planner may fail when it acts on the predictions. Chapter 64 evaluates decisions, Chapter 65 develops experimental design, and Chapter 67 examines failure modes and model exploitation.

This chapter is about how to tell these cases apart. It is in Part XI: Evaluation and Understanding. The previous chapter, "Perceptual and Generative Evaluation" (Chapter 62), handled the question of whether generated observations look right and whether their distribution matches the data. Here we develop state, constraint, and intervention diagnostics within the world-model evaluation ladder established in Chapter 61, without treating those diagnostics as nested requirements.

The chapter is organized around four evaluation families:

• State probing and representation quality. When you freeze a representation and fit a small readout, what can it recover, and what does that recovery prove?
• Object permanence and temporal consistency. Does the model maintain a coherent hidden trajectory through occlusion, and for how long?
• Physical rule and constraint tests. Does the model respect the conservation laws and boundary conditions of the regime it claims to model?
• Interventions, counterfactuals, and causal fidelity. Under a change to one mechanism, does the model predict the right change, or merely fit the observational distribution?

The four families examine separate properties. Probing measures accessibility to a readout; occlusion tests measure behavior over hidden windows; constraint tests check a named rule; interventions test a specified change under an identification design. None universally subsumes the others. Running them separately makes their different failure modes visible.

State what information the model received, what information the baseline received, and what the metric denominator counts. Compare methods on a named population, and distinguish differences in modelling from differences in information access. A privileged-state baseline can be useful, but its advantage over an observation-only method cannot be attributed entirely to better modelling.

The first three families use a point mass moving in the unit square with reflecting walls, an occluder band, and noisy position observations. The causal section then switches to a separate one-step structural model so that its confounding mechanism is explicit. The point-mass state lives in $\mathbb{R}^4$ as

$$s_t = (x_t, y_t, v^x_t, v^y_t)$$

where:

• $x_t, y_t$: position components in metres.
• $v^x_t, v^y_t$: velocity components in metres per second.

We observe only the position components through $o_t$, and the dynamics are the deterministic map

$$s_{t+1} = f(s_t, a_t)$$

defined in the simulator below, where $a_t$ is the acceleration action.

The reflecting-wall simulator returns one next state given state and action; its random initial conditions, action sampling, and observation noise are separate draws. A structural model with a specified disturbance likewise returns one outcome once all its inputs are fixed. The later causal example introduces a shared disturbance $u$ in such a one-step model. Reusing that disturbance across counterfactual branches is an explicit coupling assumption, not a property inferred from passive observations.

The point-mass simulator fits in a few dozen lines of NumPy and lets us inspect probe, occlusion, and constraint diagnostics against known truth. Neither it nor the later causal model is a benchmark for trained foundation models. Treat them as controlled fixtures: their simplicity makes each diagnostic's target and assumptions visible, without establishing its performance on a larger system.