Datasets, Benchmarks, and Experimental Design › Reproducibility and Compute Reporting

Everything above concerns whether a comparison is valid. This section concerns whether it is auditable: whether a reader could, in principle, check it.

A world-model study should be reportable as a compact, checkable protocol. The following list is organized so that each item answers a question a skeptical reader would ask. Reading it as a checklist misses the point; reading it as the set of questions a reviewer will ask is closer. Every item on the list corresponds to a plausible source of an unsupported claim, and covering them is what turns a persuasive paper into a checkable one.

Data and environment.

• Dataset or environment name, version, and content hash. For D4RL-style datasets, which variant and which generation procedure.
• Realized split manifest, or a fully specified deterministic split generator with artifact version, ordering, and randomization settings. A fraction alone is insufficient, but a fraction plus a complete generator can be reproducible.
• Grouping keys and the held-out target, with independence or dependence assumptions stated separately.
• Collection policy: who or what generated the data, at what stage of training, with what exploration, under what safety constraints.
• Provenance, licensing, and consent where applicable, without drawing legal conclusions.

Model and training.

• Training and validation budgets in environment steps, gradient steps, and wall-clock.
• Preprocessing fit boundaries: which transformations were fit on which rows.
• Initial-state and task distributions.
• Number of independent runs, and the seeds, disclosed as reproducibility settings.
• Hyperparameter search record: the space searched, the number of configurations, and the selection criterion.
• Checkpoints, so that reported numbers can be traced to a specific artifact.

Evaluation.

• Termination and reward-timing conventions: when the episode ends, when reward is credited, and how truncation is handled.
• Metrics, aggregation rule, and interval method, including any normalization anchors.
• Raw per-run scores, not only the aggregate. Aggregates can conceal per-run variation and other information needed to audit the analysis.
• Exclusions: which runs were dropped, why, and sensitivity to including interpretable recorded outcomes. If an attempted run has no valid outcome, report that fact and the declared failure-handling rule rather than inventing a score.
• The number of leaderboard submissions, if the benchmark has one.

Limitations. An explicit statement of which axes were not varied, which claims the design cannot support, and what the intended deployment regime is.

Pineau et al. (JMLR 2021) formalized much of this as a reproducibility checklist and a challenge track, and argued that sharing code and data materially improves auditability. The important caveat is that shared code is not a correctness guarantee. Rerunning flawed code reproduces a flawed claim with high fidelity.

Reproducibility aids numerical verification and auditing; it guarantees neither correctness nor external validity. A repeatable result can preserve a mistaken split or leaked fitting boundary. Some audits, such as checking a derivation or a documented protocol contradiction, do not require rerunning the full experiment. Henderson et al. (2017 preprint, AAAI 2018) documented variability across seeds and implementations in deep RL and reporting practices that hindered comparison. State what was reproduced and what that check does not establish.

The word "compute" can conceal different accounting units. Separate them.

• Environment steps. Actions sent to the real or simulated environment. This is the currency of sample efficiency.
• Environment frames. Individual rendered or physics-substepped frames. With action repeat $R$, one environment step corresponds to $R$ frames. Reporting frames while claiming steps (or vice versa) silently inflates or deflates sample efficiency by a factor of $R$.
• Evaluation rollouts. Full episodes run for measurement. These consume environment steps too, and whether they count against the method's budget must be stated.
• Model queries. Count single-candidate, one-step model-transition evaluations during planning or imagination. One rollout pass through $K$ candidate sequences of full length $H$ with one model member uses $K \times H$ such evaluations, assuming no early termination or reused prefixes. With $I$ full optimization passes and $E$ separately evaluated model members, the corresponding count is $I E K H$; extra reward or value heads need their own accounting if their costs matter. Batching can combine many evaluations into one function call. These counts are accounting proxies, not latency measurements: batching, state size, parallelization, and hardware determine actual time.
• Training cost. Gradient steps, tokens, and wall-clock, with hardware and numerical precision stated.

A static operation budget can be computed from a disclosed design without measuring runtime. Here the hypothetical planner replans at environment steps $0,10,\ldots,190$, evaluates 256 candidate sequences of length 15 at each decision, and executes the first ten selected actions before replanning. Assume one complete rollout pass with one model member at each decision, no reused candidate prefixes, and a full 200-step episode with no early termination. The block counts single-candidate one-step model-transition evaluations, not necessarily Python calls: one batched call may evaluate many candidates. The arithmetic remains conditional on those assumptions, and no controller rollout or timing measurement is performed here.

Distinguish measured GPU-hours, energy, FLOPs, memory, or latency from analytical counts and estimates. An estimate can be auditable if its method, assumptions, and uncertainty are given; do not present it as observed experimental timing. For measured wall-clock, disclose device, precision, batch size, and software stack. For latency, also define whether its unit is a decision, environment step, or episode, and whether preprocessing and planning are included.

If you compute a static budget from an algorithm, label it as such. "Under our disclosed planning configuration, each decision requires $K \times H = 3{,}840$ model transitions" is a checkable arithmetic statement. "Our method uses 4.2 GPU-hours per run" is a measurement claim that requires a measurement.

Timings produced on a CPU notebook, including any that appear in this chapter, are local illustrations only, and only where explicitly measured and labeled. Nothing in this chapter should be read as a hardware benchmark.

Suppose the split matches the claim, adaptive preprocessing respects the fitting boundary, the comparison uses a declared pairing design, uncertainty is reported, and resource accounting is clear. You can now assess the evidence for the stated estimand under disclosed assumptions. Checklist completion does not guarantee that a broad claim is true or that the comparison has enough precision to resolve it.

You do not know why the model behaves as it does. A held-out error measures discrepancy on the evaluated sessions. That discrepancy alone does not distinguish model misspecification from irreducible outcome variation, or identify which component limits performance. It does not tell you whether the encoder discarded the relevant variable, whether the transition model is over-smoothing, whether the planner is exploiting a model artifact, or whether the latent state failed to retain a memory that the task requires. Those are component-level questions, and they need component-level instruments: probing, latent ablations, attention or attribution analysis, and targeted counterfactual tests.

The next chapter, Interpretability and World-Model Debugging, introduces instruments for investigating those internal causes. A sound evaluation makes the observed contrast interpretable within its scope; it does not establish its mechanism. Causal conclusions from interpretability tools require explicit identification assumptions and supporting evidence. Controlled interventions are one useful route; identified observational inference or a justified structural derivation can provide another.