Benchmark Contamination › Code Implementation
Language AI HandbookAll chapters Benchmark Contamination › Code ImplementationOne section · Sections in this chapter5 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's implement contamination detection methods, building on our understanding of n-grams from Part II.
Setup and Data Preparation
First, we'll create synthetic benchmark and training data to show detection:
# Synthetic benchmark examples
benchmark_examples = [
"The capital of France is Paris, known for the Eiffel Tower.",
"Mitochondria are the powerhouse of the cell, generating ATP.",
"def factorial(n): return 1 if n <= 1 else n * factorial(n-1)",
"Water boils at 100 degrees Celsius at standard pressure.",
]
# Synthetic training corpus (contaminated and clean)
training_corpus = [
# Contaminated: exact match
"The capital of France is Paris, known for the Eiffel Tower.",
# Contaminated: paraphrased
"Paris, famous for the Eiffel Tower, is the capital city of France.",
# Clean: unrelated
"Machine learning models require large amounts of training data.",
# Clean: related but distinct
"London is the capital of the United Kingdom.",
]Benchmark examples: 4 Training documents: 4 Benchmark 1: The capital of France is Paris, known for the Eiffel Tower.... Benchmark 2: Mitochondria are the powerhouse of the cell, generating ATP....
These examples stand for the synthetic benchmark questions we will analyze. The first covers geographic knowledge about Paris, while the second tests biological understanding of cellular organelles. These diverse examples will help show how different contamination detection methods perform across factual domains. The synthetic nature of this data allows us to control exactly what contamination exists. This gives ground truth for validating our detection methods.
N-gram Overlap Detection
We implement character-level n-gram extraction and matching, which catches both verbatim and lightly edited contamination:
import re
def extract_ngrams(text, n=13):
"""Extract character n-grams from text."""
# Normalize: lowercase, remove extra whitespace
text = re.sub(r"\s+", " ", text.lower().strip())
return {text[i : i + n] for i in range(len(text) - n + 1)}
def compute_contamination_score(benchmark_text, training_corpus, n=13):
"""Compute what fraction of benchmark n-grams appear in training."""
bench_ngrams = extract_ngrams(benchmark_text, n)
# Collect all training n-grams
train_ngrams = set()
for doc in training_corpus:
train_ngrams.update(extract_ngrams(doc, n))
if not bench_ngrams:
return 0.0
overlap = bench_ngrams.intersection(train_ngrams)
return len(overlap) / len(bench_ngrams)
# Analyze each benchmark example
results = []
for i, bench in enumerate(benchmark_examples):
score = compute_contamination_score(bench, training_corpus, n=13)
results.append((i, score, bench[:50]))N-gram Contamination Analysis (n=13): -------------------------------------------------- Example 1: 100.00% overlap (CONTAMINATED) Text: The capital of France is Paris, known for the Eiff... Example 2: 0.00% overlap (CLEAN) Text: Mitochondria are the powerhouse of the cell, gener... Example 3: 0.00% overlap (CLEAN) Text: def factorial(n): return 1 if n <= 1 else n * fact... Example 4: 0.00% overlap (CLEAN) Text: Water boils at 100 degrees Celsius at standard pre...
Example 1 shows 100% contamination due to an exact match with training data. This shows effective detection of verbatim copying. The remaining examples show 0% overlap, correctly identifying them as clean. Notice that Example 1's paraphrased version in training (Paris, famous for the Eiffel Tower...) does not contribute to the n-gram score because no 13-character substring matches verbatim. This confirms both the method's strength (precise detection of exact copies) and its weakness (paraphrases evade detection entirely). In real evaluations you would run this analysis at multiple values of , from 5 to 13, to catch both short verbatim fragments and complete question copies.
MinHash for Scalable Detection
For large-scale detection, MinHash gives efficient approximate matching that catches near-duplicates including light paraphrases:
def get_minhash(text, num_perm=128, n=5):
"""Create MinHash signature from text using word n-grams."""
m = MinHash(num_perm=num_perm)
# Use word n-grams instead of characters for better semantic matching
words = text.lower().split()
for i in range(len(words) - n + 1):
ngram = " ".join(words[i : i + n])
m.update(ngram.encode("utf-8"))
return m
# Create LSH index for training documents
lsh = MinHashLSH(threshold=0.5, num_perm=128)
# Add training documents to index
train_hashes = []
for i, doc in enumerate(training_corpus):
m = get_minhash(doc)
train_hashes.append(m)
lsh.insert(f"train_{i}", m)
# Query with benchmark examples
minhash_results = []
for i, bench in enumerate(benchmark_examples):
m = get_minhash(bench)
matches = lsh.query(m)
if matches:
# Compute actual Jaccard similarity with best match
best_match = matches[0]
match_idx = int(best_match.split("_")[1])
jaccard = m.jaccard(train_hashes[match_idx])
minhash_results.append((i, jaccard, len(matches)))
else:
minhash_results.append((i, 0.0, 0))MinHash Similarity Analysis: ---------------------------------------- Example 1: Jaccard=100.00%, Matches=1 (HIGH SIMILARITY) Example 2: Jaccard=0.00%, Matches=0 (LOW SIMILARITY) Example 3: Jaccard=0.00%, Matches=0 (LOW SIMILARITY) Example 4: Jaccard=0.00%, Matches=0 (LOW SIMILARITY)
MinHash identifies Example 1 as highly similar to training documents (high Jaccard similarity), while correctly assigning low similarity to the clean examples. The LSH index returns candidates for both the exact match and the paraphrase, because both share enough 5-word n-grams with the benchmark question to exceed the 0.5 threshold. This confirms MinHash effectively detects near-duplicate content at scale, which makes it ideal for preliminary contamination screening across billion-document corpora, though it may miss highly abstracted semantic paraphrases that share no word sequences.
Embedding Similarity Detection
For semantic paraphrase detection, we use sentence embeddings that map meaning into a dense vector space:
model = SentenceTransformer("all-MiniLM-L6-v2") # Load embedding model
def compute_embedding_similarity(benchmark_texts, training_corpus):
"""Compute maximum cosine similarity between benchmark and training."""
bench_embeddings = model.encode(benchmark_texts)
train_embeddings = model.encode(training_corpus)
# Normalize for cosine similarity
bench_norm = bench_embeddings / np.linalg.norm(
bench_embeddings, axis=1, keepdims=True
)
train_norm = train_embeddings / np.linalg.norm(
train_embeddings, axis=1, keepdims=True
)
results = []
for i, bench_vec in enumerate(bench_norm):
# Compute similarities with all training docs
similarities = np.dot(train_norm, bench_vec)
max_sim = np.max(similarities)
best_match_idx = np.argmax(similarities)
results.append((i, max_sim, best_match_idx))
return results
embedding_results = compute_embedding_similarity(
benchmark_examples, training_corpus
)Embedding Similarity Analysis: ---------------------------------------- Example 1: Max similarity=1.000 with train_0 (CONTAMINATED) Example 2: Max similarity=0.138 with train_1 (CLEAN) Example 3: Max similarity=0.071 with train_0 (CLEAN) Example 4: Max similarity=0.095 with train_0 (CLEAN)
The results show Example 1 has high semantic similarity with the training documents due to both exact and paraphrased versions being present, correctly identifying it as contaminated. Critically, the paraphrase ("Paris, famous for the Eiffel Tower, is the capital city of France") receives a high similarity score even though it shares no 13-grams with the benchmark question. This is the embedding method's unique value: it detects semantic overlap that n-gram matching cannot see. Clean examples show lower scores, validating the threshold selection, though Example 4 about boiling water may also show moderate similarity to the London capital question due to shared geographic register.
