Deduplication › Deduplication

Training a large language model costs tens of millions of dollars and months of compute. The decisions you make about your training data determine whether that investment produces a capable model or an expensive mistake. One of the most impactful decisions is deduplication: removing copies and near-copies of documents before they enter the training set.

The intuition seems straightforward. If your corpus contains the same Wikipedia article scraped from five hundred different mirror sites, the model will see that content five hundred times. It will memorize it rather than learn from it. The model's parameters, instead of encoding broadly useful language patterns, become increasingly optimized to reproduce those repeated documents verbatim. Duplicate-heavy training data creates problems beyond storage: it degrades generalization, inflates benchmark scores without improving real-world capability, and causes models to hallucinate by confidently reproducing memorized text in inappropriate contexts.

The research community has quantified these effects precisely. Lee et al. (2022) found that training on a single-deduplicated version of C4 versus an 8-times-deduplicated version produced measurable differences in memorization rates, perplexity, and downstream task performance. The LLaMA models trained on deduplicated data consistently outperformed alternatives on standardized benchmarks despite using fewer total tokens, because unique tokens carry more information per training step than repeated ones. The practical conclusion is unambiguous: deduplication is not an optional cleanup step but a core investment in training effectiveness.

Deduplication sits at the fourth step of the data curation pipeline. After web crawling collects raw pages and document extraction strips boilerplate HTML into clean text, you have a corpus with structure and content but potentially massive redundancy. Deduplication cleans that redundancy out so that quality filtering and toxicity removal in subsequent steps operate on a corpus where every document earns its place on its own merits rather than on the merits of how many copies of it exist.

This chapter covers the full spectrum of deduplication strategies: exact hashing for perfect copies, Jaccard similarity over character shingles for near-duplicates, suffix array-based substring deduplication for repeated passages within otherwise unique documents, MinHash for scalable near-duplicate detection, and the systems engineering challenges that arise when you need to apply these methods to trillions of tokens on hundreds of machines.