Benchmark Contamination › Summary

Benchmark contamination is a basic measurement challenge in language AI. As models train on increasingly broad internet corpora, the assumption that evaluation data remains unseen becomes untenable without rigorous verification. The integrity of our entire field depends on solving this measurement problem.

We have explored contamination in its three main forms: verbatim copying, semantic paraphrasing, and conceptual overlap. Each form requires different detection methods and poses different philosophical questions about what constitutes unfair advantage. Verbatim contamination is clearly problematic and detectable. Conceptual contamination sits in a gray zone that challenges our definitions of learning versus memorizing.

Detection methods span the spectrum from precise but brittle to flexible but ambiguous:

• N-gram analysis at $n \geq 13$ gives definitive evidence of verbatim copying, with the contamination score $C(B, T) = |N_B \cap N_T| / |N_B|$ measuring the fraction of benchmark n-grams present in training.
• MinHash with LSH lets scalable near-duplicate detection across internet-scale corpora, estimating Jaccard similarity without exhaustive comparison.
• Embedding similarity catches paraphrased contamination that lexical methods miss, by measuring directional alignment in semantic space.
• Differential perplexity offers a training-data-agnostic approach when clean reference models are available, comparing $\text{PPL}_{\text{reference}}(x) - \text{PPL}_{\text{suspect}}(x)$ to identify anomalously familiar examples.
• Membership inference uses model outputs as indirect evidence of training set membership, adapting privacy-focused techniques to the contamination detection problem.

Mitigation requires a multi-layered defense: aggressive deduplication during data curation, canary insertion for definitive detection, dynamic benchmark renewal to stay ahead of training data, private holdout evaluation servers to prevent leakage at the source, and careful statistical analysis to distinguish true contamination from coincidental overlap.

The basic tension remains: we want to train on as much data as possible to build capable models, yet we need uncontaminated evaluations to measure that capability accurately. Resolving this tension, through better detection methods, cleaner data curation, and continuous benchmark renewal, is needed for the scientific integrity of language AI research. Without trustworthy evaluation, we cannot distinguish real understanding from advanced memorization, nor can we guide the field toward models that generalize rather than recite. The next chapter on benchmark saturation examines what happens when benchmarks remain static too long, as contamination and exhaustion of test signal combine to undermine their discriminative power.