HumanEval › The pass@k Metric

The primary metric for HumanEval is pass@k, which measures the probability that at least one of $k$ samples from the model passes the test suite for a given problem. This metric shows a practical reality: we rarely expect perfect code on the first attempt, and modern coding assistants often generate multiple suggestions for you to choose from. Because we cannot sample infinitely from the model, we estimate this probability using a finite number of samples $n$ (where $n \geq k$).

The metric addresses a subtle but important statistical issue. If we simply reported the fraction of correct samples ($c/n$), we would underestimate a model's utility in scenarios where users can examine multiple options. For example, if a model generates 100 samples and 10 pass, the single-sample accuracy is 10%, but the probability of finding a working solution when allowed to see 10 samples is substantially higher than 10%. Pass@k quantifies this cumulative probability, capturing the model's capacity to generate diverse, correct solutions within a search budget.

For a given problem, suppose we generate $n$ samples and find that $c$ of them pass all tests. If we were to randomly select $k$ samples without replacement from these $n$, the probability that at least one is correct is given by the hypergeometric distribution:

$$\text{pass@k} = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}$$

where:

• $n$: total number of samples generated per problem (the sample pool size)
• $c$: number of correct samples (passing all tests)
• $k$: number of samples selected for evaluation (the budget of attempts)
• $\binom{n}{k}$: the total number of ways to choose $k$ samples from $n$
• $\binom{n-c}{k}$: the number of ways to choose $k$ samples from the $n-c$ incorrect ones

The formula uses the complement rule: we calculate the probability that all $k$ selected samples are incorrect (the ratio of binomial coefficients) and subtract this from 1 to get the probability that at least one is correct. This approach assumes sampling without replacement, which is appropriate when selecting $k$ distinct samples from the $n$ generated candidates. Without replacement is the correct model here because in practice, we would present $k$ distinct solutions to you, not the same solution $k$ times.

In practice, researchers often generate $n = 200$ samples per problem but report pass@k for various $k$ values (typically $k \in \{1, 10, 100\}$). When $n$ is large relative to $k$, we can approximate the binomial coefficient ratio. We can expand the ratio of binomial coefficients step-by-step:

$$\begin{aligned} \frac{\binom{n-c}{k}}{\binom{n}{k}} &= \frac{(n-c)!}{k!(n-c-k)!} \cdot \frac{k!(n-k)!}{n!} && \text{(expand definitions)} \\ &= \frac{(n-c)!}{(n-c-k)!} \cdot \frac{(n-k)!}{n!} && \text{(cancel } k! \text{ terms)} \\ &= \prod_{i=0}^{k-1} \frac{n-c-i}{n-i} && \text{(write as product of } k \text{ terms)} \end{aligned}$$

When $n \gg k$, each term $\frac{n-c-i}{n-i} \approx 1 - \frac{c}{n}$, so the product becomes $\left(1 - \frac{c}{n}\right)^k$. This yields the approximation:

$$\text{pass@k} \approx 1 - \left(1 - \frac{c}{n}\right)^k$$

where:

• $c/n$: the empirical probability that any single sample is correct (the maximum likelihood estimate of the true pass rate)
• $(1 - c/n)^k$: the probability that all $k$ samples are incorrect, assuming independence

This approximation treats the sampling as approximately independent (with replacement), which is valid when $n$ is large and $k$ is small relative to $n$. The approximation is useful because it avoids computing large binomial coefficients and gives intuitive insight: the probability of failure decays exponentially with $k$ when the base success probability is fixed.

The final pass@k score is the mean pass@k across all problems in the benchmark, giving a single number that characterizes model performance:

$$\text{pass@k}_{\text{overall}} = \frac{1}{N} \sum_{i=1}^{N} \text{pass@k}_i$$

where:

• $N$: the number of problems in HumanEval ($N = 164$)
• $\text{pass@k}_i$: the pass@k score for the $i$-th problem

This macro-averaging approach gives equal weight to each problem, regardless of difficulty. An alternative would be to pool all samples across problems, but averaging per-problem scores prevents easy problems from dominating the metric and gives progress on hard problems equal weight. The choice of macro-averaging has an important implication: a model that perfectly solves 82 problems and fails all remaining 82 reaches the same pass@1 as a model that has a 50% success rate on every problem, even though these stand for qualitatively different capability profiles.