Data Curation and Synthetic Data › Deduplication, Filtering, and Balancing

Once you can measure quality and diversity, the mechanical work begins. A useful default order is deduplicate, filter, then balance: copies otherwise distort empirical thresholds, and balancing before filtering can amplify corrupted episodes. The order is not universal. For example, source-specific filters may need to run before cross-source deduplication, and protected rare regimes may need separate thresholds before any global pass. Record the order because each step changes the statistics seen by the next.

Duplicates are the most common and most underestimated defect in robot and agent datasets.

• Exact duplicates arise from replay-buffer re-logging, retried uploads, and resumption after a crash. They are trivial to detect with a hash over the episode's bytes, but they are rarely the interesting case.
• Near-duplicates require a task-dependent similarity rule. A robot standing still for a minute while a camera logs at 30 Hz produces 1,800 highly correlated transitions. Repeated grasps or narrowly randomized simulations can likewise form large similarity clusters. Such clusters may dominate an unweighted empirical loss while adding less independent information than their file count suggests. "Effective sample size" has several formal definitions, so use it only with a stated estimator rather than treating it as a property that directly drives gradient descent.

Near-static segments place extra weight on approximately identity transitions. Under a capacity-limited model and an average one-step loss, that weighting can bias predictions toward small changes at the expense of less common motion. Whether rollouts or subsequent collection become conservative depends on the planner, uncertainty treatment, and data-collection loop; it is a failure mode to test, not a guaranteed chain of events.

Practical detectors, roughly in increasing order of cost:

1. Exact hashing. A tuple of a few rounded state and action values, or a hash of the serialized episode.
2. Summary-statistic bucketing. Discretize episode-level features (net displacement, action range, first state, duration, spectral content) and hash the bucket. Cheap, catches the standing-still family immediately.
3. MinHash over sets of action shingles, or SimHash over fixed-dimensional sequence features. MinHash estimates set overlap; SimHash supports similarity search for vectors. Neither identifies contiguous subsequences unless sequence structure is first encoded in shingles or features.
4. Embedding-space nearest neighbors. Encode each episode with a documented sequence representation, then flag pairs below a calibrated distance threshold for removal or downweighting. Effectiveness and cost depend on the encoder, index, and dataset size.
5. Perceptual hashing of frames. For vision-heavy data, a pHash or a learned frame fingerprint catches scene-level duplication, which is often distinct from action-level duplication.

A useful rule is to inspect duplicates at more than one granularity. Duplicate episodes and repetitive transitions are different problems: an episode with a unique outcome may still contain hundreds of waiting transitions. Transition-level novelty scores or residuals from a preliminary model can support downsampling, but a high residual can also mark corruption and a low residual can mark a common transition that still matters for calibration. Validate any pruning rule against per-regime downstream error.

Filtering flags episodes that survived deduplication but may not be trustworthy for a specified use. Candidate signals include roughness under an assumed dynamics prior, residuals from a reference model, forward-backward tracker consistency, signal-to-noise estimates, numeric finiteness, and checks against known bounds such as joint limits. These signals can reveal logging or sensor faults, but legitimate rare dynamics can also look unlike the rest of the dataset.

Two cautions are worth stating plainly:

• Aggressive filtering creates covariate shift. If your quality score penalizes jerky motion, you will remove precisely the aggressive, dynamic episodes that a controller needs in order to learn what happens near actuator limits. The model then becomes confident and wrong exactly where it matters. The fix is not to loosen the filter uniformly but to filter conditionally: apply a strict quality bar within the routine regime and a looser bar (or none) in regimes you know are intrinsically rough, such as contact, recovery, or high-speed maneuvers.
• Relative and absolute thresholds answer different questions. Quantiles or robust scale estimates adapt to a dataset's numerical range but always retain a fixed fraction, even when the whole dataset is bad. Absolute limits encode external knowledge but require calibration across sensors and embodiments. In practice, combine a relative rule with validated physical or instrumentation bounds when those bounds exist.

After deduplication and filtering you often have a distribution that is clean but wrong: several hundred episodes of the robot walking and nine episodes of it turning. Balancing reshapes that distribution toward a target.

The three mechanisms are not interchangeable:

• Reweighting. Keep every episode but assign each a loss weight $w_i = p_{\text{target}}(z_i) / p_{\text{data}}(z_i)$, where $z_i$ is a regime label, $p_{\text{target}}$ is the desired regime distribution, and $p_{\text{data}}$ is the empirical regime distribution. This requires support overlap: if $p_{\text{target}}(z)>0$ where $p_{\text{data}}(z)=0$, reweighting cannot create evidence. Large weights can produce high-variance gradients. Clipping them trades variance for bias and should be treated as a tuned estimator choice, not a universal rule.
• Resampling. Draw episodes with replacement so the empirical distribution matches the target. Simple and stable in the sense that every batch looks balanced, but oversampling a 9-episode regime eightfold creates near-duplicates of those nine episodes, which is deduplication's problem reintroduced through the back door.
• Stratified or group-robust objectives. Replace or augment average loss with per-group constraints or a worst-group objective. This can emphasize a scarce predefined group, but the result depends on group definitions, regularization, and optimization; naive group DRO need not improve worst-group generalization.

Whatever mechanism you choose, state what $p_{\text{target}}$ means. It might approximate the deployed policy's query distribution, a safety-weighted distribution that deliberately overrepresents hazards, a uniform diagnostic distribution, or a mixture of these. No single choice is correct for every objective. Writing the target and evaluation weights down makes resampling and loss weighting auditable, although it does not mechanically determine filter thresholds or resolve uncertainty about deployment.