HumanEval › Benchmark Contamination and Leakage

One of the most significant concerns about HumanEval performance claims is data contamination. The HumanEval problems were designed to be novel in 2021, but as the benchmark grew in prominence, solutions appeared in blog posts, tutorials, GitHub repositories, and Stack Overflow discussions. Models trained on internet data crawled after HumanEval's release may have been exposed to the exact problems and solutions during pre-training or fine-tuning. This exposure creates an optimistic bias: the model may perform well not because it can reason about novel code generation tasks, but because it has memorized the solutions.

Detecting contamination is a methodological challenge. You cannot simply check whether the training data contains HumanEval problems verbatim: paraphrasing, reformatting, or presenting the solution as part of a tutorial discussion can expose a model to the answer without creating an exact string match. Some researchers have examined contamination by checking whether model-generated solutions resemble canonical solutions more closely than independent implementations would, but this is imperfect because elegant solutions to simple problems naturally look similar regardless of whether the model saw the reference.

A practical response to contamination concerns has been the development of private evaluation sets held by organizations and not released publicly. The MultiPL-E benchmark, for instance, extends HumanEval to 19 programming languages: since it was created after HumanEval's release, fewer solutions exist publicly in each language. This gives somewhat more contamination-resistant measurements. The Mostly Basic Programming Problems (MBPP) benchmark was released before being made public as a test set. This keeps the test problems themselves had not circulated online.

The contamination problem has also motivated a shift toward measuring capability rather than just performance. Instead of asking only whether a model passes HumanEval tests, researchers now analyze the types of errors models make, the strategies models use when they fail, and whether models can explain their reasoning. A model that has memorized solutions would be expected to fail on simple problem variants that require adapting a known solution to a new constraint, whereas a model with real understanding should handle such variants gracefully.