Deduplication › Key Parameters

Understanding the key parameters lets you tune deduplication appropriately for your corpus and use case.

• k (shingle size): Controls the granularity of near-duplicate detection. Values of 5 to 10 work well for character-level shingles. Smaller values increase false positives because short substrings appear frequently in any text. Larger values increase false negatives because long substrings are specific to particular phrasings. For short documents (under 200 characters), smaller $k$ values risk high false-positive rates; for long documents, larger $k$ values are safe.
• threshold ($\tau$): The Jaccard similarity threshold above which two documents are considered near-duplicates. Typical range is 0.7 to 0.8 for conservative deduplication and 0.5 to 0.6 for aggressive. Lowering the threshold removes more documents but risks removing materially distinct content. Raising it keeps more content but misses more near-duplicates. There is no universally correct value; the right threshold depends on the corpus and the application.
• normalization: The text normalization steps applied before hashing or shingling. Must be consistent across the entire corpus. Inconsistent normalization is a common source of deduplication failures, where the same document is kept in multiple variants because different preprocessing steps produced slightly different normalized forms.
• duplicate removal strategy: Whether to keep the first occurrence (greedy), keep the highest-quality version, or apply source-based priority. For most LLM training pipelines, keeping the highest-quality version is preferred, which requires computing quality scores before or alongside deduplication.