HumanEval › The Origins of HumanEval: Codex and the Code Generation Era

To appreciate why HumanEval was designed the way it was, it helps to understand the context in which it appeared. In 2021, OpenAI released Codex, a model fine-tuned from GPT-3 on a large corpus of publicly available code from GitHub. Codex demonstrated remarkable ability to complete code from natural language descriptions, form the basis of GitHub Copilot, and generate entire functions from docstring prompts. This was a qualitative leap from earlier code generation approaches that relied on template-filling or grammar-based synthesis.

But how capable was Codex, really? The question proved surprisingly difficult to answer. Early informal evaluations showed that Codex could produce impressively correct-looking code for simple problems, yet fail silently on edge cases that a human would catch immediately. Researchers needed a systematic benchmark that could produce a single comparable number while still capturing the nuances of functional correctness. They could not simply ask human judges to rate code quality at scale; that would be too expensive and inconsistent. They could not use BLEU scores for the reasons described above. They needed execution.

The Codex paper introduced HumanEval alongside the model, serving the dual purpose of showing the model's capabilities and establishing a reproducible evaluation protocol. Codex achieved 28.8% pass@1 on HumanEval in its initial evaluation, meaning that roughly one in four randomly generated solutions passed all test cases on the first attempt. This number was simultaneously impressive, far beyond what previous code generation approaches could reach, and sobering: it meant that more than 70% of first-attempt solutions were incorrect. The pass@100 score of 72.3% told a different story: when the model could generate 100 samples per problem, the probability of finding at least one working solution was much higher, suggesting that Codex possessed the relevant knowledge but required multiple attempts to express it reliably.

The contrast between pass@1 and pass@100 immediately raised an important practical question: what is the right $k$ for your use case? A developer using a coding assistant interactively might tolerate examining 3 or 5 suggestions. An automated test repair system might be willing to generate 100 candidates and run them all through a test suite. The pass@k metric's ability to capture performance at different budgets made it immediately useful for a range of deployment scenarios.