HumanEval › Interpreting pass@k Scores in Context

As HumanEval became the standard benchmark for code generation, a progression of models demonstrated rapid improvement. Codex achieved 28.8% pass@1 in 2021. GPT-4 achieved over 67% pass@1 by 2023. Models specifically fine-tuned for code generation, such as Code Llama and DeepSeek Coder, pushed pass@1 above 80%. By 2024, models with larger context windows and instruction tuning were approaching 90% pass@1 on the original benchmark, representing near-saturation.

This rapid saturation illustrates both the progress of the field and the limitations of any single benchmark. When models approach ceiling performance, the benchmark loses discriminative power: differences between top models become statistically insignificant given HumanEval's 164-problem sample size. The standard error of a pass@1 estimate is roughly $\sqrt{p(1-p)/164}$, which at $p = 0.9$ equals about 2.3 percentage points. This means that two models with reported pass@1 scores of 87% and 90% are not statistically distinguishable on this benchmark alone without additional testing.

The figure above quantifies this uncertainty: in the saturation regime above 85%, the 95% confidence interval spans roughly 4-5 percentage points. This means that a paper claiming model A (87%) beats model B (90%) based on HumanEval alone should be viewed skeptically unless confirmed by additional benchmarks or materially more test problems.