HumanEval › HumanEval: Functional Code Generation Evaluation with Pass@k

How do we know if a language model can write code? The goal is code that runs correctly and passes test cases, not code that merely looks right. This question became urgent as models like GPT-3 demonstrated the ability to generate plausible-looking Python functions, yet often produced subtle bugs that rendered them useless in practice. The stakes are high. Incorrect code can crash systems, introduce security vulnerabilities, or silently produce wrong results. Traditional NLP evaluation metrics like BLEU or ROUGE, which we explored in Part LIII: Evaluation Fundamentals, measure lexical overlap between generated and reference text. These metrics were originally designed for machine translation and text summarization, where n-gram overlap correlates reasonably well with human judgments of quality. But code is fundamentally different from natural language: a function can be lexically distant from a reference implementation while being perfectly correct, or lexically similar while containing a critical flaw. A single misplaced semicolon or off-by-one error can make code completely non-functional, even if 99% of the tokens match the reference.

HumanEval, introduced by OpenAI in 2021 alongside the Codex model, addressed this evaluation gap by focusing exclusively on functional correctness. Rather than comparing strings, HumanEval executes generated code against unit tests. It consists of 164 hand-crafted Python programming problems, each with a function signature, docstring, several test cases, and a canonical solution. The benchmark quickly became the standard for evaluating code generation capabilities, establishing a methodology that subsequent benchmarks like MBPP would extend and refine. Before HumanEval, the field lacked a rigorous way to verify whether models could produce working software, leading to inflated claims based on superficial metrics.

The core innovation of HumanEval lies in its evaluation metric: pass@k. Instead of measuring similarity, pass@k estimates the probability that at least one out of $k$ independently generated samples passes all test cases for a given problem. This shifts the focus from single-shot perfection to the model's ability to generate correct solutions within a budget of attempts, much as a programmer tries multiple approaches until one succeeds. This probabilistic framing acknowledges the inherent stochasticity of large language models, where temperature sampling produces diverse outputs, and a failed first attempt does not indicate incapability.