Data Curation and Synthetic Data › Data Mixtures, Provenance, and Contamination

The final stage turns curated sources into a training distribution. Mixture choices can shift performance between regimes, while leakage across a split boundary makes the evaluation optimistic and can hide regressions. The size of either effect is empirical; it should be measured rather than assumed.

Suppose you have $K$ sources, each with its own distribution $p_k$ over trajectories, and you train on

$$\begin{aligned} p_{\text{mix}} &= \sum_{k=1}^{K} \lambda_k p_k \\ \lambda_k &\ge 0 \\ \sum_{k=1}^{K} \lambda_k &= 1 \end{aligned}$$

where:

• $K$: the number of data sources being mixed
• $p_k$: the distribution of trajectories from source $k$
• $\lambda_k$: the mixture weight assigned to source $k$, a non-negative scalar with $\sum_k \lambda_k = 1$
• $p_{\text{mix}}$: the resulting mixture distribution over trajectories

The weights $\lambda$ are hyperparameters that change the population risk being optimized. They do not change the formal model class, but they can change which fitted solution the training procedure favors. Three practical approaches to setting them are:

• Proxy sweeps. Train smaller proxy models under candidate mixtures and compare them on a regime-stratified development set. This is useful only when proxy rankings predict the full training run; check that assumption on a few mixtures before relying on it.
• Direct weight fitting. Estimate development loss as a function of the mixture, then optimize the fitted response surface for a stated target risk. The response model and extrapolation range become additional assumptions to validate.
• Domain balancing heuristics. Equalize transitions per regime, target a desired per-regime contribution, or upweight groups with poor development performance. These rules are simple, but what they equalize encodes the objective.

Sampling and loss weighting can target the same expected objective when their probabilities and weights are matched correctly. Their finite-batch gradient variances depend on the losses, sampling scheme, and normalization; neither is uniformly lower variance. Mixture weights should be revalidated when source distributions, deployment goals, or the model class change.

Every episode should carry metadata answering: where did this come from, collected by what, with which version of the code, under which license, and what processing steps has it been through? Concretely, the provenance record for a world-model dataset typically includes:

• Source identity: real robot, specific simulator build, human video, learned generator, and the generator's checkpoint hash.
• Collection context: date, location, operator, policy version, task specification, and the seed for any randomized component.
• Processing history: every filter, deduplication pass, resampling operation, and transformation applied, in order, with parameters.
• License and consent: critical for human video, and often forgotten until a model ships.

Provenance supports reproducibility, regression tracing, and answers to questions such as "did we train on this episode?" It also enables contamination audits: without source identifiers and transformation history, mechanical split checks cannot reconstruct how an example entered the dataset.

Contamination means evaluation data, or information derived from it, has influenced training or model selection in a way the reported protocol does not allow. For world models, leakage can occur at several correlated units: transitions, trajectories, collection sessions, scenes, embodiments, and simulator configurations.

Consider the naive split: shuffle all transitions and assign 90% to train, 10% to test. Adjacent transitions from one trajectory are strongly dependent and can share observations, so this split does not measure generalization to new trajectories. An episode-level split is better for that question but still does not measure generalization across sessions, scenes, robots, or calibrations when those factors appear on both sides. A model can exploit shared nuisance variation and score well without transferring to a new setting.

Design the split around the factor you want the model to generalize across:

• If you care about generalization to new scenes, split by scene, and ensure no scene appears on both sides.
• If you care about generalization across days or calibration drift, split by collection session.
• If you care about generalization across embodiments, split by robot or by simulator configuration.
• If you care about generalization from simulation to reality, hold out reality entirely.

Then verify identifiers and content mechanically rather than trusting folder structure. Exact IDs and hashes catch direct reuse; perceptual hashes can flag near-duplicate frames; nearest-neighbor searches in a documented embedding can surface suspicious cross-split similarity for review. A small distance is not by itself proof of leakage, because legitimate repeated states can be close.

There is a second contamination channel through synthetic generation. If a generator is fitted using evaluation examples and then emits training samples, information from the evaluation set has influenced training even if no generated sample is byte-identical to it. Hold out evaluation data before fitting any generator used for training data, and keep generator selection and tuning on the development split.

Finally, note the interaction between contamination and curation: a filter tuned on the evaluation set is itself a leak. If you choose your roughness threshold by looking at which threshold maximizes held-out rollout accuracy, you have fit the threshold to the test set, and the reported number is optimistic. Tune filters on a development split, and keep a third split untouched until the end. This three-way split, train on train, tune on dev, report on test, is a small amount of extra discipline that pays off every time you need to trust your own numbers.