Deduplication › Understanding the Deduplication Pipeline End-to-End
Language AI HandbookAll chapters Deduplication › Understanding the Deduplication Pipeline End-to-EndOne section · Sections in this chapter10 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
To see how all these components fit together, consider the pipeline used for a production LLM training dataset at the billion-document scale.
The first step is document-level exact deduplication using MD5 hashing on normalized text. This pass is extremely fast (hashing one billion documents takes a few hours on a small cluster) and removes all perfect or near-perfect copies. Because it is so cheap, it is always worth running this step first; it reduces the input size for the more expensive near-duplicate detection pass.
The second step applies MinHash LSH near-duplicate detection at the document level with a threshold around 0.7. This pass is the computationally expensive one: computing MinHash signatures for one billion documents requires significant distributed compute, and the LSH bucketing and within-bucket comparison steps require careful implementation to avoid memory bottlenecks. In practice, this pass runs on a Spark cluster over several hours to a day.
The third step applies union-find to cluster near-duplicates and select representatives. With quality scores computed in an earlier quality filtering step, this selection step retains the highest-quality version of each cluster. Documents below a quality threshold are discarded even if they would have been the first representative in a greedy approach.
The fourth step, applied only in pipelines targeting high data quality, is substring deduplication using suffix arrays or token-level ngram matching. This step is the most expensive and most complex; some pipelines skip it entirely and rely on the combination of exact and near-duplicate deduplication to achieve sufficient cleaning.
The final step is an audit: checking the deduplication results by sampling random clusters, verifying that clustered documents are true near-duplicates, and spot-checking whether any materially distinct documents were incorrectly grouped. This manual validation is essential because the parameters (threshold, shingle size, MinHash parameters) interact in complex ways with the specific corpus, and only empirical validation can confirm the pipeline is working as intended.