Data Curation and Synthetic Data › Limitations & Impact

Every technique in this chapter is a heuristic that trades one kind of error for another, and none of them is free.

• Curation optimizes proxies. Roughness, prediction residuals, cluster counts, and nearest-neighbor distances do not by themselves establish downstream performance. A dataset can score well on aggregate data-quality metrics while missing an action-alignment defect or control regime. When the intended consumer is a planner, validate material curation choices by retraining the model and measuring planner outcomes as well as data statistics.
• Filters have a horizon problem. A filter tuned for one-step residuals can favor episodes already handled well by the preliminary model, removing difficult but valid regimes along with corruption. Conditional filtering can use looser thresholds in intrinsically rough regimes, but those regime definitions are themselves modeling assumptions that require validation.
• Balancing can conflict with deduplication. As the worked example showed, resampling a scarce regime up to parity repeats a handful of episodes and changes their loss weight without creating new support. Generating new samples with a simulator avoids exact repetition but imports simulator bias instead. Choose the trade-off by measuring per-regime downstream error and calibration.
• Synthetic data adds information from its generator, not new empirical evidence about the deployed system. A simulator, procedural generator, or learned rollout model can make assumptions explicit and cheaply explore them. Its mixture weight should depend on measured real-data coverage, simulator validation, and the cost of the target error; plentiful real data does not by itself imply a zero synthetic weight.
• Contamination can inflate evaluation. Transition-level random splits share trajectory context across train and test, while episode-level splits can still share scene, session, or calibration factors. Report the split unit and the generalization factor it is designed to test.
• Provenance and licensing constrain which data may be used and how results can be reproduced. For human video, record source, consent basis, license, processing history, and applicable retention restrictions when the data enters the pipeline; reconstructing that record later can be difficult or impossible.

Together, these techniques let you state and test what evidence a model receives. Deduplication changes repeated examples' weight; filtering trades suspected corruption against selection bias; balancing changes the target risk; simulation adds controllable but assumption-laden trajectories; and provenance makes those choices traceable. None guarantees competence outside the training support. They make the limitations measurable, which is the prerequisite for using the decision-centric models discussed in Part VIII: Decision-Centric Research Lineages.

The natural next question is what happens when data does not stop arriving. Real fleets produce a continuous stream, and a curated static dataset is a snapshot of a moving target. Distribution shift, replay, forgetting, and the choice between training from scratch and updating in place are the subject of Online, Offline, and Continual Learning.