Deduplication › Summary

Deduplication is one of the most impactful data curation decisions you will make for a language model training pipeline. The key takeaways are:

• Exact deduplication uses cryptographic hashing to find bit-for-bit identical documents. Normalization before hashing is essential to catch formatting variants, and this step is so fast and cheap that it should always be applied first.
• Near-duplicate detection uses Jaccard similarity over character n-grams (shingles) to find documents that are similar but not identical. The Jaccard threshold, the shingle size, and the normalization scheme all affect which documents are flagged and must be chosen carefully for the specific corpus.
• Substring deduplication uses suffix arrays to find repeated passages within otherwise unique documents, addressing memorization driven by passage-level repetition rather than document-level duplication. This is the most computationally demanding deduplication method but addresses a distinct class of redundancy that document-level methods miss.
• Scalable deduplication requires MinHash signatures and Locality-Sensitive Hashing to avoid the $O(N^2)$ cost of pairwise comparison. The next chapter covers MinHash in full detail, including the mathematical derivation, LSH banding, and practical implementation.
• Union-find provides an efficient data structure for clustering near-duplicate pairs into groups and selecting a representative from each cluster, ideally the highest-quality member.
• Deduplication interacts with quality filtering: the order of these steps and the strategy for selecting cluster representatives both affect the final data distribution. In practice, quality scores should inform which cluster representative is kept.
• False positives exist: aggressive deduplication can remove legitimate content that happens to share common vocabulary or phrasing, potentially creating coverage gaps in the training corpus. Evaluation against a labeled sample of document pairs is essential to verify that the pipeline is calibrated correctly.

The choice of deduplication strategy involves balancing memorization reduction against information loss, and that balance depends on the model scale, the downstream tasks, and the nature of the corpus. Understanding these trade-offs is what separates a thoughtful data curation pipeline from a naive one. The algorithms in this chapter provide the technical foundation; calibrating them to your specific corpus requires empirical investigation and domain knowledge about where duplication arises and what forms it takes.