Datasets, Benchmarks, and Experimental Design › Datasets, Benchmarks, and Experimental Design

Consider a hypothetical robotics study. A group logs four hours of teleoperation from one arm, fits a small next-joint-state predictor, and randomly assigns 70% of the transitions to training and 30% to testing. Suppose it reports a mean absolute error of 0.004 radians. Its draft claims that the model transfers to new collection sessions. A separate, stronger claim would be transfer to an arm it has never seen. The hours, split, and error in this opening are illustrative, not results from a reported experiment.

A reviewer asks whether the test contained a new session. It did not; nor did it contain a new arm. Both partitions came from the same logged collection. A payload or calibration offset shared within that collection can create reusable context, but sharing a constant does not make every state and action pair correlated. Nearby training transitions may help a model infer that context; the split alone does not prove that it did. Even if the illustrative 0.004 error were measured correctly, it would describe performance on the chosen held-out rows. It would not by itself establish transfer to a new session. It would also not establish transfer to a new arm.

That gap, between the question a number answers and the question a claim asks, is the subject of this chapter.

The World-Model Evaluation Ladder organized five complementary protocols: one-step prediction, open-loop rollout, intervention-response comparison, closed-loop decision quality, and transfer. The three following chapters examined perceptual and generative fidelity; state, physics, causality, and memory; and planning, control, and policy evaluation. They supplied instruments for different questions, not interchangeable certificates. This chapter asks whether the experiment producing a number supports the sentence you want to write about it.

That is a different skill, and it is not learned by reading more metrics. It is learned by interrogating the design: which unit was held out, what was fit on which rows, how many independent replicates exist, what the interval means, and what compute was spent. A perfectly reasonable metric applied to a badly designed comparison produces a perfectly precise number attached to a claim it cannot support. We will see several concrete versions of that failure in this chapter.

In most empirical sciences, a number is only as good as the design that produced it, and world-model research compounds this because the objects being measured (learned dynamics, learned abstractions of physical structure, planners that query the model thousands of times per decision) are themselves complicated artifacts whose "correctness" depends on the deployment regime. A model that generalizes perfectly within one lab's robot may fail catastrophically on the same robot after a firmware update, and a model that fails to generalize at all may still be the best available option if the deployment target is exactly the collection regime. Neither of those statements can be evaluated by staring at a loss curve. They require naming the estimand, selecting the units and splits that isolate it, and reporting the resulting numbers in a form a skeptical reader can check.

We keep the notation from Planning, Control, and Policy Evaluation: $s_t$ is the true state, $o_t$ the observation, $a_t$ the action, $f$ the true transition function, $\hat f$ a learned dynamics model, $r(s_t, a_t)$ the current-step reward, $\pi$ a policy, $T$ the episode length, $H$ the planning horizon, $K$ the number of candidate action sequences per decision, and $G$ the finite-episode return. A few new symbols appear here:

• $j$ indexes a session, a collection grouping unit. A session can be one teleoperation run or one logging period. Different sessions may still share a robot, operator, or scene; group disjointness does not by itself establish independence.
• $c_j$ is a persistent, session-constant quantity (an offset, a calibration, a payload) that the learner is not given.
• $M$ is the number of transitions and $W$ the number of lagged states used as features, so a feature matrix has shape $(M, W+1)$ once the action column is appended.
• $n$ is the number of independent replicates, $D_i$ a paired difference, $\sigma_D$ its standard deviation, and $\delta$ a target difference worth detecting.

The dynamics toy observes the physical scalar directly: $o_t = s_t$. Its session constant $c_j$ is not observed. The complete Markov state is therefore $(s_t, c_j)$, with $c_j$ unchanged during an episode. Relative to that complete state, the learner is partially observed even though there is no visual perception problem. History can supply information about the persistent context, so context inference and belief over unobserved state are not categorically separate problems. This toy isolates one static-context setting; it does not evaluate general perceptual robustness or memory architectures. Contextual Markov Decision Processes provide a related formal treatment of episode-level context.

The chapter proceeds in four movements. First, what a benchmark is, and why counting benchmarks is a poor substitute for mapping claims to required coverage. Second, leakage and shift: how to name the unit you are holding out, and what happens when you do not. Third, the statistics of comparison: replication, pairing, intervals, and prospective power. Fourth, reporting: a study protocol, and honest compute accounting. The recurring question is always the same. Does this comparison support the stated claim?