Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Benchmark Contamination › Worked Example: Detecting Contamination

Michael BrenndoerferMarch 5, 20262 min read

Language AI HandbookAll chapters

Part 54 of 67Benchmark Evaluation

Chapter 421Benchmark Contamination › Worked Example: Detecting Contamination

Reading section 4 of 12 · Worked Example: Detecting Contamination · 2 minSection 4 of 12 · 2 min

Section 4 of 12 · 2 min read

Benchmark Contamination › Worked Example: Detecting ContaminationOne section · Sections in this chapter4 / 12 ↓

Sections in this chapter12

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's walk through a concrete example of detecting contamination in a question-answering benchmark. Consider this benchmark question:

"What is the powerhouse of the cell? A) Nucleus B) Mitochondria C) Ribosome D) Golgi apparatus"

If this exact string appears in the training data, we have verbatim contamination. But suppose the training data contains:

"Students often ask: which organelle is the powerhouse of the cell? The answer is the mitochondria, not the nucleus or ribosome."

An n-gram analysis with $n=5$ would partially catch this paraphrase ("powerhouse of the cell" as a 4-gram matches, but the longer context differs). However, embedding similarity would flag high semantic overlap between the two texts. N-gram analysis at $n=13$ would miss this entirely, since no 13-token sequence appears verbatim in both. This example illustrates why we need multiple detection strategies: the n-gram method catches direct copies but misses rephrasing, while embeddings catch conceptual similarity even across lexical variation.

Now consider a code example from HumanEval:

In[6]:

Code

def factorial(n):
    if n == 0:
        return 1
    return n * factorial(n - 1)

If training data contains the same logic but with different variable names (x instead of n) or as an iterative loop instead of recursion, n-gram matching fails entirely. We need either abstract syntax tree (AST) matching or neural-based code similarity detection. AST matching normalizes variable names and ignores comments, then compares the structural tree representing the program's logic. Two functions that compute the same result through equivalent logic produce similar ASTs even when their surface text differs completely.

This case shows a broader principle: each data modality (text, code, math) requires specialized contamination detection adapted to that modality's structure. The methods that work for natural language prose do not transfer directly to code or mathematical expressions.