HumanEval › Summary

HumanEval changed code generation evaluation by measuring functional correctness rather than lexical similarity. The benchmark consists of 164 hand-crafted Python problems, each with test cases that unambiguously determine whether a generated solution works correctly. The pass@k metric captures the probability that at least one of $k$ samples passes all tests, calculated as:

where:

• $n$: total samples generated per problem
• $c$: count of correct samples (passing all tests)
• $k$: number of samples selected for evaluation

This approach acknowledges that language models are probabilistic: even capable models may not generate correct code on the first attempt, but given multiple tries, their effective problem-solving rate increases substantially. The metric's formulation using the hypergeometric distribution properly accounts for sampling without replacement, while the approximation $1 - (1 - c/n)^k$ gives intuition about the exponential decay of failure probability with increased sampling budget.

Key points to carry forward:

• HumanEval evaluates functional correctness through code execution, not lexical similarity, representing a basic move from BLEU-based evaluation
• The pass@k formula uses the complement rule with the hypergeometric distribution; the approximation $1 - (1-c/n)^k$ applies when $n \gg k$
• Temperature affects the diversity-quality tradeoff: lower temperatures improve pass@1, higher temperatures improve pass@k at large k
• The benchmark's 164 problems give limited statistical power, especially in the saturation regime where top models cluster above 85%
• HumanEval+ revealed that many solutions declared correct by the original sparse tests fail on more complete test suites
• Data contamination from public solutions is a significant concern when comparing results across model generations

The benchmark's legacy lies in the specific numbers it produces, in establishing execution-based evaluation as the standard for code generation research, and in popularizing the pass@k framework that quantifies probabilistic correctness in stochastic generation systems.