HumanEval › Worked Example

Let us walk through a concrete example to solidify the calculation. Suppose for a specific problem, we generate $n = 5$ samples and find that $c = 2$ of them pass all tests.

To calculate pass@1:

$$\begin{aligned} \text{pass@1} &= 1 - \frac{\binom{5-2}{1}}{\binom{5}{1}} \\ &= 1 - \frac{3}{5} \\ &= 0.4 \end{aligned}$$

This matches our intuition: with 2 correct samples out of 5, the probability of randomly selecting a correct one is $2/5 = 0.4$.

To calculate pass@2:

$$\begin{aligned} \text{pass@2} &= 1 - \frac{\binom{3}{2}}{\binom{5}{2}} \\ &= 1 - \frac{3}{10} \\ &= 0.7 \end{aligned}$$

There is a 70% chance that at least one of two randomly selected samples is correct. This is substantially higher than the 40% single-sample probability. This shows the benefit of multiple attempts.

For pass@3:

$$\begin{aligned} \text{pass@3} &= 1 - \frac{\binom{3}{3}}{\binom{5}{3}} \\ &= 1 - \frac{1}{10} \\ &= 0.9 \end{aligned}$$

With only 3 incorrect samples available, selecting 3 samples without replacement has only a 10% chance of getting all incorrect ones. This leaves a 90% chance of getting at least one correct solution. As $k$ approaches $n$, the probability approaches 1. This shows the fact that with enough attempts, we will eventually examine all correct solutions in the pool.

Notice what happens at the boundary. When $k = n = 5$, we must select all 5 samples, so we are guaranteed to include the 2 correct ones. Pass@5 equals exactly 1.0. This behavior is automatically handled by the formula: $\binom{n-c}{k} = \binom{3}{5} = 0$ since $k > n-c$, so the ratio is 0 and pass@5 = 1. When $c = 0$, all samples are incorrect, so no matter how many we select, none will be correct. Pass@k = 0 for all $k$ when $c = 0$. These boundary cases give useful sanity checks when implementing the metric.