Datasets, Benchmarks, and Experimental Design › Limitations & Impact

This framework constrains claims; it does not guarantee results or manufacture missing evidence. Splitting logs from one robot does not create observations from a second robot. An empirical new-robot claim therefore needs new-robot evaluation or a separately justified transport argument with stated structural assumptions. Narrow an unsupported empirical claim or collect the missing evidence rather than treating an aggregation or interval as a substitute.

The second limitation concerns assumptions rather than a blanket failure of uncertainty methods. The paired Student-$t$ interval has exact nominal coverage for iid Gaussian differences with positive population variance and $n\geq2$. With nonnormal differences its finite-sample coverage need not be exact, especially when few runs reveal little about the distribution. The percentile bootstrap uses only the observed runs and cannot infer unseen population tails from extra resamples. The normal sample-size formula is approximate planning, not a guarantee for an unknown heavy-tailed population. State the assumptions, report uncertainty and practical effect sizes, and leave conclusions unresolved when the evidence does not distinguish the relevant alternatives.

An aggregate error increase on new sessions can be compatible with several mechanisms: loss of reusable context, a changed population of dynamics parameters, changed initial-state sampling, or limited fitting data for the model class. One aggregate contrast need not identify which mechanism produced it. Targeted controlled comparisons can investigate those alternatives; the earlier State, Physics, Causality, and Memory Evaluation chapter and the next chapter provide relevant instruments, not automatic causal certificates.

Aligned held-out units and uncertainty reporting improve cross-paper interpretation when populations, metrics, protocols, and resource constraints are also aligned, or their differences are accounted for. A well-designed unresolved comparison can bound what the evidence currently distinguishes, but it does not prove equality or intrinsic task difficulty. A flawed comparison can still yield diagnostics; it should not support the claim its design cannot identify. Rigorous design also has costs: reserving groups reduces fitting data relative to using every group, independent runs add compute, and prospective power calculations may show that a proposed budget cannot resolve the target effect.

"Your split was random" is not by itself a refutation; ask which estimand it targets and whether its dependence and information access match the intended use. A random transition split can be appropriate for a specified interpolation claim. Do not use it as sufficient evidence for new-unit transfer, and do not dismiss it solely because it is random.

Finally, this chapter deliberately stops short of two adjacent topics. It does not analyze why models fail internally, which belongs to the next chapter, and it does not analyze how models are exploited by planners and optimizers, which belongs to Failure Modes and Model Exploitation. Both are places where a valid experimental design can still lead you to a wrong conclusion about a system, and both deserve their own treatment.