Datasets, Benchmarks, and Experimental Design › Train-Test Leakage and Dataset Shift

In the hypothetical opening, the draft explicitly claimed transfer to new sessions while its split evaluated rows from the same logged collection. The mistake is a mismatch between that transfer claim and the estimand, the quantity the reported number estimates. Mean absolute error can be useful; a suitable metric does not make an unsuitable held-out unit answer the intended question.

Three roles need to be distinguished before any split is drawn, because they have different rules.

• Training. Rows used to fit parameters and data-adaptive preprocessing for this experiment. Fixed transforms and independently pretrained representations need separate provenance disclosure; they are not necessarily refitted on these rows.
• Validation (model selection). Rows used to choose hyperparameters, architectures, checkpoints, early stopping, or analysis rules. They are no longer an untouched test of the selected procedure. Their descriptive scores remain meaningful; subsequent inference requires new data or a method that accounts for adaptivity.
• Final test. Rows evaluated after model and analysis choices are frozen. If their results guide a new hyperparameter, checkpoint, or metric choice, they have participated in selection. Favorable selection on noisy results can produce optimistic performance estimates; the untouched-test interpretation no longer holds.

For example, choosing the best of five observed seeds and reporting it as representative hides the selection. Unusual outcomes are not by themselves grounds for excluding a run. Prespecified validity criteria or independently confirmed corrupt execution can justify exclusion, but report every attempted run, reasons, and sensitivity to the exclusions. Ordinary training failures that belong to the procedure being evaluated should not disappear from that procedure's estimand. The choice of five in this example is illustrative, not a required run count.

A good habit is to make the report include every seed you ran, even the ones you did not like, with a short reason for each if any were excluded. When the reader sees all five seeds and where they landed, they can do the analysis you would do if you were reviewing yourself. When the reader sees one seed and a rationale about outliers, they can only trust or distrust, and trust is not the currency of empirical claims.

Before you can hold anything out, you must decide what the independent unit is. Candidate units in world-model research include:

• transitions $(s_t, a_t, s_{t+1})$
• episodes or trajectories
• collection sessions (one teleoperation run, one logging day)
• subjects, patients, or participants
• objects, tools, or manipulated items
• robots, vehicles, or embodiments
• scenes, rooms, or maps
• procedural levels or game seeds
• tasks or objectives
• dates, sites, or institutions

Each candidate corresponds to a different claim. Holding out transitions supports claims about interpolation. Holding out trajectories supports claims about new trajectories. Holding out subjects supports claims about new people. The unit is not a technical detail you choose after the fact; it is the formalization of which thing the claim is about. Two groups can run "the same" experiment with the same data and get different numbers, not because one is right and one is wrong, but because they held out different units.

Here is the rule that governs everything else in this section:

The unit you hold out must match the generalization claim.

For direct empirical tests, hold out transitions for the declared within-session target, sessions for new-session transfer, and robots for new-robot transfer. A structural argument for transporting evidence is a different route and needs its own assumptions. No split label supplies that argument automatically.

A random transition partition can estimate a specified within-session interpolation target when its information access and joint dependence match the intended use. Matching the marginal distribution of individual rows is not enough: deployment on a fresh session can have the same row marginals but independent session context. The design must also distinguish retrospective interpolation from forward forecasting. Reporting a random-split result with that scope is not evidence of misconduct.

It becomes misleading when it is used to support claims about a different estimand: new trajectories, new sessions, new embodiments, or new environments. The mechanisms that make it misleading are specific and worth naming:

• Shared latent context. A persistent session-level quantity (calibration, payload, operator style, scene lighting) is present in both partitions. A sufficiently flexible model can exploit it.
• Overlapping history windows. Adjacent windows containing $W$ consecutive state observations share $W-1$ of those observations. The fraction of all input features shared depends on $W$ and any additional features, such as actions. A test row may therefore share observations with a training row.
• Near duplicates. A controller holding a pose or a looping policy can produce repeated or near-identical states. Check their presence in the actual log rather than assuming a prevalence.
• Future information. Whole-trajectory normalization, smoothing, or embeddings can depend on observations unavailable at the prediction moment. A fixed known constant need not; trace each transform's inputs and time boundary.
• Common data sources. Two sessions may share a base model, a base scene, or a base policy, so holding one out does not hold out the underlying source.

Shared latent context can be missed by checks that only compare row IDs. A correctly implemented row split can leave training and test rows from the same session. If session context affects the observed relation, the model has an opportunity to exploit it; the split alone does not prove it did. Whether that opportunity is legitimate depends on the claim. It can be appropriate for within-session interpolation and insufficient for new-session transfer.

Temporal dependence alone is not proof of label leakage, and time indexing alone does not force autocorrelation. Suppose scalar losses $Z_1,\ldots,Z_M$ are covariance-stationary, with finite positive variance $\sigma_Z^2$ and lag correlation $\rho_k$. Expanding the variance of their average gives

$$\operatorname{Var}(\bar Z)=\frac{\sigma_Z^2}{M} \left[1+2\sum_{k=1}^{M-1}\left(1-\frac{k}{M}\right)\rho_k\right].$$

Here $M$ is the number of observed losses, and $\rho_k$ is the correlation between losses $k$ positions apart. There are $M-k$ pairs at lag $k$, which produces the factor $1-k/M$. When the bracket is positive, an effective sample size relative to independent equal-variance losses is $M_{\mathrm{eff}}=M$ divided by that bracket. Positive aggregate lag correlation reduces this effective size; a negative aggregate can increase it. Treating positively dependent losses as independent can therefore make intervals too narrow. The formula concerns the loss sequence, rather than the correlation of raw states.

Leakage is a different question: did prediction-unavailable information, or information excluded by the declared held-out-unit protocol, enter fitting or prediction? Trace feature times and fitting provenance for that check. Use dependence-aware inference for correlated outcomes. Grouped or chronological splits can enforce a target boundary, but do not by themselves eliminate every remaining dependence.

Kapoor and Narayanan catalogued leakage patterns across machine-learning-based science and argued that many published results are affected. Their taxonomy is a useful diagnostic vocabulary. Rephrased for world models:

• Task mismatch / mis-specified estimand. The split is valid for one question but reported for another. This is the opening story.
• Correlated observations. Rows can be dependent within a unit. Treating positively dependent losses as independent can overstate effective sample size. Shared nuisance context can also carry information across partitions.
• Preprocessing leakage. Scalers, imputers, embeddings, feature selectors, target encoders, augmentation statistics, deduplication thresholds, and PCA bases fit on all rows including test rows.
• Target leakage. Features that encode the target, or that are only available after the prediction moment.
• Duplicate or window overlap. The same transition, or heavily overlapping windows, appear in both partitions.

The first category is a target/reporting mismatch; the others describe data or information relationships that need separate checks. Better preprocessing alone does not change a mismatched target. Grouping original trajectories before constructing their windows prevents windows from the same trajectory crossing partitions, provided duplicate parent trajectories are handled too. Use the taxonomy to propose diagnostics, not as a guarantee that a fix has been found.

Split raw groups first. Then fit every learned transformation on training rows only, and apply the frozen transform to validation and test.

This includes learned standardization, imputation, principal components, feature selection, vocabulary construction, target encoding, augmentation statistics, and data-adaptive deduplication thresholds. Fitting them on all rows lets test information influence fitting and violates the declared inductive protocol. The score effect is model-dependent; it is not guaranteed to be optimistic or small. For example, invertible affine feature scaling leaves unregularized least-squares predictions unchanged when an intercept is fitted and the augmented training design has full column rank. Without that identifiability condition, the fitted training values remain unchanged, but independently chosen minimum-norm coefficients can change predictions outside the training subspace. Selecting features using test targets is a different mechanism: it can favor variables that happen to correlate with those targets. Neither the information-boundary violation nor its magnitude should be inferred from a generic transform ranking.

There is a legitimate alternative: a transductive protocol, in which the model may use unlabeled test inputs at fit time. That is a different estimand, it is sometimes the right one, and it must be disclosed. It is not the same as accidental preprocessing leakage, and it should not be described as "standard."

Two practical constructions follow from the unit decision.

Grouped splits. Build a group key (session ID, robot ID, subject ID, level seed) and partition on the key. Every window from the same trajectory stays in the same partition. This is the design used in the executable example below.

Chronological splits with an embargo. For past-to-future forecasting, train on information available before the evaluation period. Derive the gap from the observations used by each row. Under this chapter's convention, a row centered at $t$ uses $W$ state values from $t-W+1$ through $t$, and predicts every next state from $t+1$ through $t+L$. Its inclusive observation support is $[t-W+1,t+L]$. A row centered at $u$ shares an observation exactly when $|u-t|\leq W-1+L$, assuming no additional dependency. Thus last training center $a$ and first test center $b$ have disjoint supports when $a+L<b-W+1$. For $W=3,L=1$, centers three steps apart still share one observation; four steps apart do not. An embargo may remove rows on either side to enforce this condition. Removing $W+L$ steps on both sides is not a universal minimum. Wider feature, label, or preprocessing dependencies need a wider support calculation. For endpoint-only or sparse targets, use the actual support set; the interval can be a conservative bound rather than an exact description. Disjoint supports remove this mechanical overlap; they do not guarantee statistical independence or eliminate every leakage mechanism.

Grouping changes which units are held out; it need not reduce the number of test rows. An embargo removes some eligible observations, which may be training observations rather than test observations. Report row counts and independent-unit counts separately. Precision also depends on within-unit dependence, beyond the split's name. A same-session target can be easier for some models than a new-session target, but that is not a universal ordering. Choose the target first, then budget enough independent units for its uncertainty.

If new sessions are drawn from the same population law as training sessions, the protocol is grouped holdout without an imposed population shift. Finite empirical distributions can still differ. Removing shared session context can make prediction harder for a model that relied on it, but grouping does not guarantee an error increase. This tests "new session, same regime"; it is not by itself evidence about a new robot.

To test an OOD axis, specify how its population law differs. Examples include robots with changed mass or actuation limits, or levels generated with parameters outside the training range. A held-out site or date is a grouping choice; it is evidence of a particular shift only when the relevant changed factors are identified. State both the held-out unit and the varied factor for each evaluation set.

Let us build the smallest experiment that makes the mechanism visible. The true dynamics are a stable one-dimensional controlled process with a persistent session intercept:

$$s_{t+1} = A s_t + B a_t + c_j + \varepsilon_t, \qquad \varepsilon_t \sim \mathcal{N}(0, \sigma^2),$$

where:

• $s_{t+1}$: the next state of the simulated process
• $s_t$: the current state, observed directly (that is, $o_t = s_t$)
• $a_t$: the control input applied at time $t$
• $A = 0.90$: shared damping coefficient across all sessions
• $B = 0.35$: shared action-gain coefficient across all sessions
• $c_j$: the persistent intercept for session $j$, drawn once per session as $c_j \sim \mathcal{N}(0, 0.12^2)$ and held fixed across all time steps within that session
• $\varepsilon_t$: process noise, drawn independently at each step as $\varepsilon_t \sim \mathcal{N}(0, \sigma^2)$ with $\sigma = 0.02$

The learner observes the physical scalar $s_t$ directly and is never given $j$ or $c_j$. The complete predictive state includes the unobserved persistent context, so a history-based belief about $c_j$ can improve prediction. Session overlap changes the information a model can reuse; it does not guarantee that a particular model has inferred the context or that context inference is always necessary for the chosen loss.

The first code block sets up the simulator and the shared constants.

Three qualifications matter here. The observed error orderings are not guarantees for every draw. The within-session result has its own scope; presenting it as new-session transfer would change the claim. Finally, the observed gap is not a demonstrated information-theoretic barrier. With known shared coefficients and past state-action transitions, residuals $s_{k+1}-A s_k-B a_k$ give noisy observations of $c_j$. Averaging those residuals can reduce context uncertainty. The implemented feature window does not include past actions, however, and this chapter has not measured a fraction of the gap closed by that enlarged estimator. That would require its own matched evaluation.

Leakage is about information crossing a boundary it should not. Dataset shift is about the test distribution differing from the training distribution. They are related but distinct, and they need different responses.

The shifted cohort above makes the distributional change concrete. Plotting the session intercepts shows what moved.

• Covariate shift. For one specified prediction task, the full input distribution changes while the output conditional given those inputs stays fixed. A state-transition predictor uses inputs $(s_t,a_t)$; an observation predictor may use observation history and action. New lighting can change the observation domain without changing physical dynamics. New layouts are pure covariate shift only if the chosen inputs contain enough layout information for the transition conditional to remain unchanged.
• Dynamics shift. The transition law at the chosen state-action description changes, for example with friction or actuator parameters. In the toy, the conditional law given $(s_t,a_t,c_j)$ retains the same coefficients, while the context population changes; the law averaged over unobserved context can change. A fixed robust or context-conditioned system may already handle some changes. Otherwise adaptation, identification, or uncertainty-aware control are candidate responses, not universal requirements. Ordinary input normalization is not a general dynamics-shift remedy.
• Action-distribution shift. A policy change can alter the visited state-action distribution, especially when it changes reachable actions. Two policies differing only on unreachable states need not induce different occupancy. Mismatch between logged support and policy visits is a challenge in offline model-based RL, discussed in Offline and Conservative Model-Based RL.
• Task shift. The objective, reward, or termination criterion changes; specify which reward and success conventions differ.
• Embodiment shift. A different robot, sensor suite, or morphology, which can change the observation and action interfaces as well.

Match the protocol to the declared factor. Appearance randomization is one option for an appearance claim, not a universal covariate-shift treatment. Changed dynamics parameters can be evaluated zero-shot or with a declared adaptation protocol. For action-distribution shift, report collection and evaluation policies and their support. Direct empirical evaluation of new-task performance uses task-level holdout; direct empirical evaluation of new-robot performance uses robot-level holdout. A separate transfer argument can support a conditional claim under explicitly justified structural or identification assumptions; it is not a measurement on those held-out units. State whether the observation/action interface changes rather than assuming every new embodiment changes its dimensions.

The practical recommendation is to make the OOD axis a named column in your results table, not an unstated property of the test set. "Unseen session," "unseen robot," and "unseen level" are different claims, and a single "test" column hides which one you measured. A well-designed results table reads like a small matrix of estimands, and each row or column tells the reader exactly which axis was varied. When the axes are named, replication becomes easier, because the next group can reproduce the same axis even if their robot or their task is different.

When a pretrained world model's corpus is incompletely documented and an evaluation suite was publicly available before the training cutoff, evaluation material may have entered pretraining. Memorization can then be mistaken for transfer. Public availability establishes an opportunity for contamination, not that a particular model encountered the suite.

The honest treatment is a statement about uncertainty, not a verdict. Unknown pretraining membership is an uncertainty, not evidence that contamination occurred. What you can do:

• Report what the model card or technical report says about pretraining data and cutoff dates.
• Prefer evaluation suites that are versioned, dateable, or held privately.
• Where possible, evaluate on data collected after the reported pretraining cutoff.
• Report results with and without a contamination-sensitive subset if such a subset exists.
• Avoid the two extremes: do not assert contamination without evidence, and do not assert cleanliness because the model card is silent.

Repeated leaderboard submissions provide feedback on shared test data. Scores can be dependent or deterministic: submitting identical deterministic predictions twice need not produce a fresh draw. Adaptive choices can nevertheless overfit that feedback. When candidate scores estimate the same quantity with noise, selecting the maximum can produce optimism; the size depends on dependence and the selection procedure. Report submissions, adaptation, and the selection rule. A private held-back evaluation unused for those choices provides a separate test, subject to its own population and sampling assumptions.