The World-Model Qualification Test › The World-Model Qualification Test

Suppose you are handed two trained models and told that each one achieves an impressive prediction score on a robotics dataset. Model A watches a camera stream and produces startlingly realistic videos of the next few seconds. Model B takes the same camera stream plus the robot's commanded action and predicts the next frame, less beautifully, but conditioned on what the robot does. Which of these is a world model?

If your answer is "both," consider what happens when we plug each one into a controller. Model A will happily hallucinate futures for any action you feed it, because nothing in its training taught it that actions cause outcomes. Model B can be asked a question a planner needs answered: "if the gripper closes now, where will the object be?" Only one of these models can sit inside a planning loop and change the agent's behavior for the better.

This thought experiment is worth pausing on, because it exposes something uncomfortable about how machine learning models are usually judged. Both models would look excellent in a paper or a leaderboard entry. Both produce low loss values, and both generate plausible-looking outputs. Yet they belong to entirely different categories of artifact. Model A is a forecaster: it extrapolates patterns in observation sequences. Model B is a candidate world model: it encodes a relationship between decisions and their consequences. The distinction only becomes visible when we ask what the model will be used for, and this is precisely the question that standard evaluation pipelines never ask. A video-generation score cannot tell you whether closing the gripper moves the object, because that information simply is not present in an unconditional prediction task.

This chapter gives you a practical instrument for telling them apart: the world-model qualification test. It is not a benchmark with a leaderboard. It is a set of four questions you ask of any candidate model, backed by experiments you can run. The questions are:

• State sufficiency: Does the model's internal state carry everything needed to predict the future?
• Action conditioning: Does the model treat actions as causes, not as decoration?
• Counterfactual validity: Do predictions hold for actions and situations the training data never contained?
• Decision usefulness: Does an agent that plans with this model do better?

Each criterion exists because a specific, common failure mode slips past naive evaluation. We will meet each failure mode, formalize the criterion that catches it, and then run all four tests on two competing models of the same physical system. By the end you will have a reusable experimental template alongside the vocabulary.

It is worth setting expectations about how these four criteria relate to one another before we dive in. They are ordered deliberately. The first two concern what the model represents internally, which is where most failures originate. The third concerns where those representations remain trustworthy, which is a property of the interaction between the model and its training data. The fourth concerns whether any of the preceding properties translate into better behavior, which is the only outcome that ultimately matters to the system's owner. Each criterion presupposes the ones before it: a model with an insufficient state cannot be action-conditioned in any meaningful sense, and a model that fails counterfactual validity cannot be trusted by a planner no matter how good its in-distribution scores look. Reading them as a pipeline, rather than as four independent checkboxes, is the mental habit this chapter tries to install.