Datasets, Benchmarks, and Experimental Design › Benchmark Taxonomies and Coverage

We will distinguish five objects that can otherwise be conflated in a result table. These are working definitions for reading the experiment, not a claim that every research community uses the words identically.

The practical consequence: an interactive control suite is not automatically a released offline dataset, and a released offline dataset is not automatically an interactive benchmark. If your claim is about learning from logged data without further interaction, you need the dataset and its collection policy. If your claim is about closed-loop control, you need the environment and the protocol that governs how many interactions each method gets.

The five definitions suggest questions to ask before opening a result table:

• What was the input to fitting, and can it be inspected?
• Is the model allowed to query an environment, and under what budget?
• What objective and termination define success?
• How was the split chosen?
• How was the metric chosen?
• How was the aggregation rule chosen?
• What was compressed away to produce the leaderboard number?

Trace the answers to the experiment's artifacts and protocol. An undisclosed choice can leave the claim's support uncertain; an identified mismatch can show that the number answers a different question.

A subtlety that trips people up: a dataset and an environment can be produced by the same team, packaged in the same release, and used together, and they can still be conceptually distinct objects. The dataset is a snapshot; the environment is a generative process. A method that performs well when training on the snapshot has not necessarily demonstrated anything about the process, and a method that performs well when interacting with the process has not necessarily demonstrated anything about the snapshot. Blurring the two can cause a claim to exceed what the measured object supports.

Before naming any particular suite, it helps to have a vocabulary for what a benchmark can and cannot cover. These dimensions are the axes along which a claim either is or is not supported.

• Observation modality. Proprietary state vectors, RGB, depth, tactile, proprioception, language, or mixtures. A model evaluated only on state vectors has not been tested on pixels.
• State accessibility. Fully observed, partially observed with a known observation model, or partially observed with an unknown one. Observation-model knowledge changes available inference assumptions; it does not require a different architecture in every case.
• Action coverage and collection policy. Which actions were taken, by whom, at what stage of training, with how much exploration, and under what safety constraints. These choices determine the observed support for action-conditioned evaluation.
• Time horizon. How many steps before the relevant event occurs. A model that is accurate for 10 steps may be useless at 500.
• Stochasticity. Deterministic dynamics, small process noise, or multimodal outcomes. A deterministic transition test does not test whether a model represents stochastic or multimodal transition uncertainty; it need not favor every deterministic model under every metric.
• Contact and physical constraints. Free-space motion versus contact-rich manipulation with friction, impacts, and non-smooth dynamics.
• Partial observability. Which decision-relevant state is unavailable to the agent. A belief or another history summary may be needed; some tasks can be solved without explicitly representing every hidden quantity.
• Task diversity. How many distinct objectives exist, and how different they are from one another.
• Out-of-distribution axes. Which factors vary between training and test: layout, appearance, object set, physics parameters, embodiment, sensor calibration, or task structure.
• Measurement tier. Whether the reported number is a prediction error, a state-estimation error, a planning success rate, or a deployment outcome on hardware. Earlier chapters built this ladder. A benchmark directly tests the tiers it measures; inference to another tier requires an explicit, separately justified structural or identification argument.

These dimensions can interact. The time resolution and event-recording scheme determine which contact events a test exposes, while a short rollout may not reach contact at all. Diverse tasks can differ in difficulty and reward scale; their sampling weights affect aggregation. The dimensions are therefore not independent checklist items. State how each relevant choice affects the event, observation, or population being measured.

Coverage utility is claim-relative: two well-tested dimensions can be enough for a narrow claim, while nine irrelevant ones may not help it. Ask whether a benchmark tests the factors your claim requires. A famous suite can be unsuitable for a specific question, and a less familiar one can be suitable. Reputation and claim fitness are different properties; this is not a claim of statistical independence between them.

It is worth looking at four widely used suites and stating, precisely, what question each was built to ask. None of them is a universal world-model benchmark, and describing them that way is a category error.

DeepMind Control Suite (Tassa et al., 2018) is a set of continuous-control tasks built on a physics simulator, with a standardized interface and reward terms that are individually interpretable. Its generally continuing, infinite-horizon tasks are distinct from the fixed 1,000-step evaluation episodes used in the original paper: that evaluation limit truncates the task rather than defining an intrinsic terminal state. Its original question was to give continuous control a consistent, well-specified task set so that algorithm progress could be compared and reward structure could be inspected rather than guessed. Its limitation is equally clear: it is an environment suite, not a released offline dataset with documented collection policies. Strong performance on its tasks is not by itself evidence of action-conditioned fidelity under a fixed logged dataset, nor evidence of generalization to unseen tasks or embodiments.

D4RL (Fu et al., 2020) is a collection of offline datasets whose defining feature is collection-policy diversity: demonstrator data, medium-quality data, mixtures, replay buffers, random data, and multitask datasets. Its original question was to make the collection regime part of the benchmark rather than an unstated detail. The limitation follows directly: a result on a D4RL dataset is a statement about that collection policy. Improving on "medium" data does not imply improving on expert data, and vice versa, because the data distributions differ in ways that change which algorithmic ideas help.

There is also a provenance caution worth internalizing. The Farama Minari D4RL documentation states that its reproduced group is not identical to all original D4RL datasets, even though it was generated using the same principles. Two artifacts with the same name and similar statistics can differ in details that matter. Version and provenance are part of the result, not metadata.

The D4RL note generalizes to any dataset with a shared name and a sharded lineage. When you cite a result on "D4RL halfcheetah-medium," you are citing a specific artifact with a specific history, and if you did not generate it yourself you are relying on the artifact being what its name suggests. This is why content hashes and explicit version identifiers matter: they turn a name into a specific, checkable object. It is a small piece of discipline that pays off every time someone tries to reproduce a number and finds that the artifact they downloaded produces a different curve.

Procgen (Cobbe et al., 2019) consists of sixteen procedurally generated game-like environments, designed to measure sample efficiency and generalization across levels. Its original question was whether a policy trained on some levels transfers to unseen levels of the same game. Those levels can vary layouts, appearances, entities, and event timing. Standard unseen levels drawn from the same generator are a held-out-level generalization test, not automatically an out-of-distribution test. A claim about changed physics, embodiment, or a new task family needs a protocol that varies those factors.

Physion (Bear et al., 2021) presents simulated physical scenarios and compares machine predictions to human predictions, including on scenarios designed to probe specific physical concepts. Its original question was how well models and people predict physical outcomes from visual scenes. The limitation is that this is a passive prediction protocol. The agent is not choosing actions to achieve a goal, so a Physion result is not automatically evidence about action selection, planning, or control.

These examples show why a suite's intended question matters. A strong result can be quoted as though it answers a different question that the protocol never measured. That is a category-drift mechanism to check for, not a measured claim about how frequently papers overclaim or evidence of fraud.

Benchmark names identify suites but do not fully specify which factors a particular experiment tested. A requirements matrix makes that scope explicit. It has claims as rows and coverage requirements as columns. To support one of these scoped empirical claims, the benchmark evidence must address every factor designated as necessary for that claim; filling required cells is not by itself a correctness guarantee.

The last row deliberately names a deployment involving every listed factor. Closed-loop operation alone does not require hidden state, contact, stochasticity, or a horizon over 100 steps. Likewise, contact is required in the third row because its claim names contact, not merely because actuation limits change. Deployment latency is a separate hardware-and-software measurement, absent from this qualitative matrix. Adding benchmarks that duplicate the same axes does not add missing axes; a benchmark that varies a new factor can. Coverage is the union of the tested factors, not the count of suite names.

That union is only a first check. A claim about a combination of factors needs evidence on the relevant joint setting, or explicit assumptions justifying transfer from separate tests. Contact tests on familiar robots plus free-space tests on new robots do not alone test contact on a new robot. Filling every required column with separate marginal tests is not a certificate for their combination.

The coverage table is a small binary matrix. A heatmap makes the missing columns visible at a glance.

The requirements matrix is also useful alongside a second matrix of actual measurements. Compare the paper's stated requirements with the factors its experimental section says it tested. A required factor absent from those measurements is a coverage gap. The binary requirements heatmap above does not itself contain benchmark results and cannot certify any of the four claims.

One well-designed relevant test can be more informative for a claim than five unspecified or irrelevant suite names. Neither one nor five is a universal benchmark-count requirement.

Once a benchmark has multiple tasks, you need an aggregation rule that defines what "performance on this benchmark" means. Different defensible rules can rank the same methods differently. Choosing a favorable rule after inspecting test results is adaptive analysis selection, even if neither fitted model changes. Label that choice exploratory or evaluate the selected analysis on fresh data.

Define the task distribution first. A claim about a fixed suite concerns that finite set of tasks. A claim about a task population also needs a specified population and sampling design. More variants of one mechanism do not imply broader mechanism coverage: eight pushing tasks and two cloth tasks assign pushing 80% of the weight in an equal-per-task average. That does not guarantee 80% of its numerical contribution when score scales differ.

Fixed-suite inference versus sampled-task inference. For a fixed suite, uncertainty can come from training runs and evaluation episodes while task identities are held fixed. A claim about a sampled task population adds task-sampling uncertainty. Its magnitude depends on task variability and the sampling design; it need not be the largest component. Treating sampled tasks as fixed omits that component and can understate uncertainty. Replicating more runs on the same tasks does not supply more independent task draws.

Average versus tail. A task mean rewards aggregate score, not necessarily competence on every task: narrow excellence can compensate for poor outcomes elsewhere. A minimum or the fraction below a declared threshold exposes different failure behavior. If deployment assigns substantial cost to rare failures, report a matching tail measure alongside the mean. The appropriate choice follows the objective rather than a preferred ranking.

Reward-scale sensitivity. Averaging raw returns across tasks with very different numerical scales gives the larger-scale tasks more influence for a comparable proportional change. A scale difference alone does not prove that a task dominates every comparison: the actual method differences and task weights also matter.

Normalized scores. Tasks with very different return magnitudes contribute unevenly to a raw mean, so a single large-scale task can dominate the aggregate. The common fix rescales each task's raw return onto a comparable scale. The normalized score is:

where:

• $R$: the method's return on the task
• $R_{\text{ref}}$: a reference anchor (often a random or weak policy), used as the low end of the scale
• $R_{\text{expert}}$: a strong reference return, used as the high end of the scale

The score expresses how far the method's return sits above the weak anchor, as a fraction of the gap between the weak and strong anchors. Three things must be stated explicitly when you use this.

1. What the anchors are. A fixed positive anchor gap preserves method ordering within one task. Different positive gaps across tasks can change an aggregate ranking by changing effective task weights. A negative gap reverses the within-task direction. Disclose the anchors and weighting convention.
2. What happens when the denominator is zero or negative. A zero gap makes this formula undefined. A near-zero gap amplifies a fixed nonzero numerator and makes the score sensitive; it does not make every score large. Specify exclusions, clipping, or an alternative rank summary before examining favorable rankings. Clipping changes the reported statistic and can hide tails.
3. Why it is not a probability. A normalized score can exceed 1 (a method can beat the "expert" reference) and can be negative (a method can be worse than random). It is not bounded in $[0,1]$ and must not be described as if it were.

Because the score can exceed 1 or fall below 0, plots drawn as if it were a probability are misleading. The distribution it summarizes is over return differences, not over event frequencies. When a figure shows box plots of normalized scores on a $[0,1]$ axis, the reader is being nudged toward an interpretation that the numbers do not support. A figure that explicitly allows scores to range past 0 and 1, with labeled reference lines at the anchors, is more honest.

A small synthetic distribution shows the point. The scores below are illustrative; they are not from the toy dynamics above.