The World-Model Qualification Test › Interpreting the Battery

Stepping back, the experiment compressed the qualification test into a few minutes of CPU time, and its untidy results are the point:

• State sufficiency was probed cheaply with a history sweep. Extra history helped, but the curve did not flatten, so none of the tested windows earned a sufficiency claim.
• Action conditioning was identified under wide random actions and failed under narrow policy data: the learned action gain reversed sign despite low in-distribution error.
• Counterfactual validity was measured as error inflation off the training distribution, and it reframed "model quality" as a property of model-plus-data coverage.
• Decision usefulness required the environment in the loop. Both learned controllers beat the blind baseline, history helped modestly, and neither matched the oracle.

Notice the division of labor across the battery. Three of the four tests ran entirely offline, on recorded data, at negligible cost, and they localized the failure precisely: to the representation, to the data coverage, or to neither. Only the fourth required interacting with the environment, which in real systems means spending hardware time, risking wear, and accepting safety exposure. This economics is the quiet argument for the battery's ordering: exhaust the cheap tests first, and arrive at the expensive one with a hypothesis rather than a shrug. Closed-loop evaluation is irreplaceable, but it is also the scarcest resource in most applied programs, and the earlier criteria exist partly to protect it.

Two habits from this chapter are worth carrying forward permanently. First, always report the noise floor or an irreducible-error baseline alongside model error, so that "good" has a reference point. Second, never let a model be graded solely by the distribution its data came from, because a planner's job is to leave that distribution.