Benchmark Contamination › Key Detection Parameters

Michael BrenndoerferMarch 5, 20263 min read
Language AI HandbookAll chapters
Part 54 of 67Benchmark Evaluation
Chapter 421Benchmark Contamination › Key Detection Parameters
Reading section 6 of 12 · Key Detection Parameters · 3 minSection 6 of 12 · 3 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 421 of 513

Section 6 of 12 · 3 min read

Benchmark Contamination › Key Detection ParametersOne section · Sections in this chapter6 / 12 ↓
Sections in this chapter12
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The key parameters for contamination detection are:

  • n (character-level): Length of character n-grams for verbatim matching. The implementation uses n=13n=13 to give strong evidence of direct copying while minimizing false positives from short phrases.
  • num_perm: Number of hash functions for MinHash signatures. Using 128 permutations balances detection accuracy with computational efficiency. More permutations reduce variance in the Jaccard estimate but increase memory usage.
  • threshold: Jaccard similarity threshold for MinHash LSH indexing. Set to 0.5 to identify near-duplicate documents while filtering out incidental overlap.
  • n (word-level): Size of word n-grams for MinHash. Set to 5 to capture real semantic units rather than individual words.
  • similarity thresholds: Classification cutoffs for contamination scores, including 0.8 for n-gram overlap, 0.85 for embedding similarity (contamination), and 0.7 for suspicion levels.

These thresholds are not universal constants. They depend on the benchmark's domain, the granularity of detection required, and the acceptable false positive rate. A benchmark containing highly common factual statements (capitals, boiling points) requires higher thresholds to avoid spurious contamination flags, while a benchmark with unusual or technical phrasing can use lower thresholds safely.

Out[17]:
Visualization
Line chart showing precision, recall, and F1 score curves as detection threshold increases from 0.5 to 0.95. Vertical dotted lines mark N-gram threshold at 0.8 and embedding threshold at 0.85.
Precision-recall trade-off for contamination detection at varying threshold values. As the threshold increases, precision rises (fewer false positives) but recall falls (more true contamination is missed). The optimal operating point depends on whether the goal is complete contamination detection or conservative flagging. Vertical lines mark the standard thresholds for n-gram and embedding methods.