Deduplication › Why Deduplication Matters More Than You Might Expect

Before diving into algorithms, it is worth understanding just how pervasive duplication is in web-scale corpora. A raw Common Crawl snapshot contains hundreds of billions of tokens, but a surprising fraction of that content is duplicated. Legal boilerplate appears on millions of pages. Navigation menus, cookie consent notices, and footer text repeat across entire domains. News articles get syndicated across dozens of outlets with identical or near-identical text. Stack Overflow answers get scraped into tutorial sites, StackExchange clones, and programming forums. Books that entered the public domain appear on Project Gutenberg, Archive.org, and hundreds of derivative sites simultaneously.

The scale of this problem is larger than most practitioners expect. When the EleutherAI team deduplicated the Pile, a 825 GB text corpus assembled from diverse sources, they found that roughly 30% of its raw content was duplicate or near-duplicate material. The RedPajama dataset preparation process applied aggressive deduplication across all seven of its component datasets and found similar rates. In some individual sources like GitHub code repositories, duplication rates can exceed 50%, because developers fork repositories, copy boilerplate license headers, and reuse utility functions across thousands of projects.

Carlini et al. (2021) demonstrated that GPT-2 would verbatim reproduce sequences from its training data when prompted with the beginning of those sequences. The sequences most likely to be reproduced were the most duplicated ones in the training set. This has serious practical implications: a model that memorizes legal terms of service or personal information scraped from web pages represents a privacy and liability risk in addition to being a technical shortcoming. From a legal standpoint, a model that can output a full copyrighted text on demand creates exposure under copyright law. From a safety standpoint, a model trained on chat logs containing personal data may reconstruct those conversations when prompted with fragments.

Duplication also distorts what the model learns to value. A language model trained on data where tech blog posts outnumber medical literature by a hundred to one will develop strong priors for tech blog prose style and vocabulary. If those tech blog posts are themselves duplicated across aggregator sites, the effective imbalance is even worse than it appears from document counts alone. Deduplication is therefore partly a data balance problem as well as a storage optimization.

There is also a subtler effect on training dynamics. Stochastic gradient descent treats each training example as an independent sample from the data distribution. When the same document appears repeatedly in the training set, the model's gradient updates reinforce that document's patterns at the expense of all other patterns. In the extreme case, a document appearing thousands of times effectively becomes the training objective for a portion of the model's parameters. The model stops being a general language model and starts becoming a specialized reproducer of that document. This dynamic is particularly dangerous because it is not reflected in training loss: a model that memorizes repeated documents can achieve low perplexity on a validation set constructed from the same corpus, yet perform poorly on truly novel text.

The relationship between duplication and benchmark performance deserves special attention. Many popular benchmarks draw from the same web sources that constitute training data. If a question-answer pair or its near-duplicate appears thousands of times in training data, the model may be scoring well on the benchmark by retrieval rather than by reasoning. Deduplication both reduces this benchmark inflation and forces models to achieve good benchmark performance through generalization to unseen examples.