Datasets, Benchmarks, and Experimental Design › Summary

A benchmark couples a task collection with a protocol. Its evidence must match the claim's target population, available information, and evaluation question. Not every fixed-target risk claim requires changing an experimental factor.

• Separate the objects. Dataset, environment, task, benchmark, and leaderboard describe different parts of an experiment. Logs without observed or recoverable actions do not directly test action response; causal intervention identification needs extra assumptions. An interactive suite is not automatically a released offline dataset.
• Coverage beats count. Map scoped claims to required factors and compare those with actual measurements. Duplicate benchmark axes do not add missing coverage; a new tested factor can. Claims about joint settings need joint evidence or justified transfer assumptions, not merely separate tests of each factor.
• Aggregate deliberately. Define the task distribution before averaging. Normalized scores need stated anchors, a rule for zero denominators, and the acknowledgment that they are not bounded in $[0,1]$. IQM and performance profiles are useful robust summaries with precise interpretations, not universal cures.
• Name the held-out unit and justify independence separately. Sessions, robots, subjects, levels, tasks, dates, and sites correspond to different generalization targets. Disjoint grouping keys alone do not prove independent samples.
• Random transition partitions are not always wrong. They can estimate a specified interpolation target when information access and dependence match deployment. They do not by themselves establish transfer to a different held-out unit. Temporal dependence alone is not label leakage.
• Split raw groups before adaptive preprocessing in the declared inductive protocol. Fit learned transforms on training rows only, disclose fixed or independently pretrained transforms, and distinguish transductive test-input use. Derive any chronological embargo from the feature and target supports.
• Holding out a group does not determine whether there is shift. New groups can come from the same population law or a changed one. Name both the held-out unit and any changed population factor.
• Transitions are not training replicates. Many transitions under one fit provide within-run observations, not new fits. Independent training replicates require stated conditioning and randomization assumptions; fresh-data claims need data-draw variation too.
• Pair, and derive why. For i.i.d. replicate pairs with finite variance and $D_i=Y_{A,i}-Y_{B,i}$, $\operatorname{Var}(\bar D)=(\operatorname{Var}(Y_A)+\operatorname{Var}(Y_B)-2\operatorname{Cov}(Y_A,Y_B))/n$. Positive covariance helps relative to independent outcomes with the same marginals; negative covariance hurts. Diverging feedback paths do not imply zero covariance.
• Average within a run first for the training-procedure comparison. Use run-level differences as its replicates. Its bootstrap resamples complete independent runs, not flattened transitions; a different conditional-risk target needs its own sampling unit.
• Plan the replication level. $n \approx ((z_{1-\alpha/2} + z_{1-\beta})\,\sigma_D / \delta)^2$ is a planning approximation, not a guarantee. Halving $\delta$ roughly quadruples $n$. No blanket three-seed or five-seed rule applies.
• Interpret intervals correctly. Overlapping marginal intervals do not settle a paired difference, and absence of significance is not equivalence. Equivalence needs prespecified bounds and a stated interval level; TOST at 0.05 uses the 90% interval. Statistical and practical significance are different questions.
• Choose baselines for their actual contrast. Simple dynamics, action-agnostic, and model-free baselines do not automatically isolate components. Controlled comparisons, identified observational analysis, or justified structural derivations can support component attribution under their stated assumptions. True-dynamics planners are privileged diagnostics, not automatic return upper bounds.
• Ablations have two meanings. Removal at inference and retraining without a component answer different questions. One-factor designs can miss interactions; an interaction-identifying design needs sufficient combinations and replication for its target.
• Report the protocol and resources. Include artifact versions and hashes, split manifests, collection policy, budgets, preprocessing boundaries, seeds, search records, raw run scores, and exclusions. Separate environment interactions, frames, rollouts, model-transition evaluations, and training cost. Label measurements, analytical counts, and estimates distinctly.
• Reproducibility is not external validity. Rerunning flawed code reproduces a flawed claim. A complete protocol makes a result auditable; it does not make it true.

The question to carry forward is: which estimand does this number estimate, and is it the target of my claim? Alignment makes the result relevant evidence. Its uncertainty and remaining assumptions still determine how far the conclusion can go.