Deduplication › Limitations and Practical Considerations

Deduplication is powerful but not a silver bullet, and the choices you make have consequences that extend beyond the deduplication step itself.

Every deduplication algorithm has false positives: documents it incorrectly identifies as near-duplicates and removes. A document that discusses a well-known fact using standard phrasing might appear similar to many other documents discussing that fact, even though each version contributes unique context, reasoning, or examples.

This is particularly acute for factual domains. Legal documents, scientific papers, and news articles often use standardized language to describe the same facts. Aggressive deduplication might remove most of the corpus's coverage of a particular topic because all discussions of that topic share common terminology and sentence structures. The resulting trained model may lack robustness on those topics: it has learned only the single surviving version of the content, rather than developing invariance across multiple phrasings and contexts.

A practical mitigation is to perform topic-aware deduplication: apply lower similarity thresholds for sources with naturally diverse language (fiction, opinion pieces, creative writing) and higher thresholds for sources with naturally formulaic language (legal documents, scientific abstracts, patent filings). This requires segmenting your corpus by source type before applying deduplication, which adds pipeline complexity but significantly improves the quality of the result.

Deduplication is agnostic to quality by default. If you have one high-quality and one low-quality version of the same content, which one you keep depends entirely on which one you process first (in the greedy approach) or on which cluster representative selection criterion you use.

In practice, you want to run quality filtering before or alongside deduplication, and use quality scores to select which member of each near-duplicate cluster to retain. Documents with higher readability scores, fewer grammatical errors, or higher source quality signals (from Wikipedia versus random web pages) should be preferred as the cluster representative. This requires that quality scores be computed before the deduplication step selects representatives, which means the quality filtering and deduplication steps are not entirely independent. The pipeline design must account for this dependency.

Web-scale corpora are crawled over months or years. A news article might appear in your corpus multiple times with different timestamps: once when it was originally published, once a week later when it was syndicated, and once a year later when it was included in a retrospective. Document-level deduplication will correctly identify these as near-duplicates, but you might want to keep the most recent version, not the first one seen.

This requires tracking metadata (publication dates, crawl timestamps) alongside document content and using that metadata to inform which cluster representative to keep. In practice, this is often implemented by assigning a priority score that combines quality signals with recency signals, then selecting the highest-priority document from each cluster.

Deduplication reduces memorization, but some memorization is desirable. A model that can accurately recall factual information, reproduce correct syntax for programming languages, or recite standard formulas is more useful than one that cannot. Aggressive deduplication can reduce this useful memorization along with the harmful variety.

The research community is still working out the optimal deduplication strategy for different model sizes and use cases. Smaller models trained on highly deduplicated corpora tend to have better generalization because they cannot afford to waste capacity on memorized content. Larger models have enough capacity to benefit from some redundancy in training data, because the redundancy reinforces important patterns rather than just memorizing them. This suggests that the optimal deduplication threshold scales with model size: very small models should be trained on heavily deduplicated data, while very large models may tolerate more redundancy without it harming generalization.

Most large training corpora assemble content from multiple sources: web crawl data, books, Wikipedia, code repositories, scientific papers, and so on. Near-duplicate detection within a single source is straightforward, but cross-source deduplication, finding near-duplicates that span different source types, is more complex.

The challenge is that different sources may have systematically different normalization and formatting conventions, making documents that contain substantially similar material appear more different than they are to a Jaccard-based detector. A Wikipedia article and a web page that mirrors the same Wikipedia content may have slightly different formatting, citation styles, and section structures. Their Jaccard similarity might be 0.5, which is below a typical deduplication threshold of 0.7, even though they express nearly identical information.

Cross-source deduplication also raises policy questions. Should you remove a web page that closely paraphrases a Wikipedia article, even if the web page adds a paragraph of original commentary? The original commentary might be low-quality, but it is unique content. The decision depends on your data quality philosophy: do you prioritize removing redundancy or preserving uniqueness?

Measuring whether your deduplication pipeline is working correctly requires building a ground-truth evaluation set. This means manually labeling a sample of document pairs as near-duplicates or not, then measuring precision (what fraction of flagged pairs are true near-duplicates) and recall (what fraction of true near-duplicate pairs were flagged).

Building this ground truth is labor-intensive but essential. Common failure modes that manual evaluation can catch include: the threshold being too aggressive for short documents (two unrelated documents that happen to be short may have high Jaccard similarity by chance), normalization inconsistencies causing the same document to appear as multiple non-duplicates, and corpus-specific idioms (e.g., programming comments like "TODO: fix this") appearing so frequently that they create false near-duplicates between unrelated code files.

A simpler diagnostic, though less rigorous, is to check the distribution of cluster sizes after running near-duplicate detection. In a well-configured pipeline, most documents should be singletons (no near-duplicates found), a substantial fraction should be in clusters of size 2 to 5, and very few should be in clusters of size 100 or more. If you see a large fraction of documents in very large clusters, your threshold may be too low, and you are grouping materially distinct documents together.