Code Evaluation › The pass@k Metric

The most important metric in code evaluation is pass@k: the probability that at least one of $k$ sampled solutions to a problem is functionally correct.

When you use a language model interactively for coding, you rarely commit to its first output without review. You might ask it to regenerate if the first attempt looks wrong, or you might use an integrated development environment that shows you multiple completions side by side and lets you choose. This realistic usage pattern motivates pass@k. If a model can solve a problem correctly on at least one of 10 tries, that is practically valuable even if it cannot solve it reliably on every first attempt.

pass@k also gives you a way to reason about model deployment. If you want to show the user one solution and be confident it is correct, you care about pass@1: the model needs to get it right the first time. If you are using the model to generate candidate patches that a human engineer will review before applying, pass@5 or pass@10 might be more relevant: you want the correct solution to appear somewhere in the candidates, so the engineer just has to recognize it rather than produce it from scratch. If you are running automated tests and accepting any solution that passes, pass@k for large $k$ captures the ceiling of what the model can achieve on that problem at all.

The metric was introduced explicitly to escape the limitations of evaluating only pass@1. Before pass@k, researchers would report the best-of-$k$ success rate: sample $k$ outputs, report 1 if any pass and 0 otherwise. This captures a similar intuition, but the estimator has high variance. If you happen to sample more or fewer correct solutions than expected by chance, your estimate shifts accordingly, making it hard to compare results reliably across papers.

The combinatorial pass@k formula provides a statistically principled way to measure the same quantity with substantially lower variance. Rather than committing to exactly $k$ samples, you generate a larger pool of $n$ samples and use the observed count of correct solutions to estimate the probability analytically. This design choice, more samples feeding an analytical estimator rather than fewer samples feeding a direct count, is what makes pass@k a reliable research metric.

Let $n$ be the total number of solutions sampled for a given problem, and let $c$ be the number of those solutions that pass the test suite. The probability that at least one of $k$ randomly chosen solutions, drawn without replacement from the $n$ samples, is correct equals one minus the probability that all $k$ chosen solutions are wrong.

The number of ways to choose $k$ solutions that are all wrong, drawing from the $n - c$ incorrect solutions, is $\binom{n-c}{k}$. The total number of ways to choose any $k$ solutions from all $n$ is $\binom{n}{k}$. So the probability that all $k$ chosen solutions are wrong is:

$$P(\text{all wrong}) = \frac{\binom{n-c}{k}}{\binom{n}{k}}$$

Taking the complement gives the pass@k estimator:

$$\text{pass@}k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}$$

where:

• $n$: the total number of samples generated for the problem
• $c$: the number of correct samples (passing all tests) among the $n$ samples
• $k$: the number of samples shown to the user or tried in production
• $\binom{n-c}{k}$: the number of ways to choose $k$ samples that are all incorrect
• $\binom{n}{k}$: the total number of ways to choose any $k$ samples from $n$

This formula is only valid when $k \leq n$. When $n - c < k$, meaning there are fewer incorrect samples than the budget $k$, it is impossible to choose $k$ samples that are all wrong, so pass@k equals 1.0 exactly. The formula handles this as a boundary case.

An alternative definition would be: sample exactly $k$ solutions, run the tests, and report whether any passed. This is simpler but has high variance, especially when $k$ is small. If you happen to sample more correct solutions than expected by chance, you overestimate the model's pass@k. If fewer, you underestimate.

The combinatorial formula above provides an unbiased estimator of the true pass@k probability given that you have already sampled $n \geq k$ solutions and observed $c$ correct ones. By generating more samples than $k$, typically $n = 200$ for careful research evaluations, and evaluating pass@1, pass@10, and pass@100 all at once, you get stable estimates without rerunning the model multiple times.

This efficiency is significant. Running a large language model 200 times per problem is a substantial compute investment, but it is a fixed cost that gives you the entire pass@k curve. You can estimate pass@1, pass@10, and pass@100 all from the same set of 200 samples, with variance that decreases as $n$ increases. This makes systematic benchmark evaluation tractable even for researchers without enormous compute budgets, as long as they commit to the investment once per model per benchmark.

Suppose a model generates $n = 10$ solutions for a sorting problem, and $c = 3$ of them pass the test suite. Let us compute pass@1 and pass@5 explicitly.

For pass@1:

$$\begin{aligned} \text{pass@1} &= 1 - \frac{\binom{n-c}{1}}{\binom{n}{1}} \\ &= 1 - \frac{\binom{10-3}{1}}{\binom{10}{1}} \\ &= 1 - \frac{7}{10} \\ &= 0.30 \end{aligned}$$

For pass@5:

$$\begin{aligned} \text{pass@5} &= 1 - \frac{\binom{n-c}{5}}{\binom{n}{5}} \\ &= 1 - \frac{\binom{7}{5}}{\binom{10}{5}} \\ &= 1 - \frac{21}{252} \\ &\approx 0.917 \end{aligned}$$

So the model has only a 30% chance of solving the problem on a single attempt, but if you pick any 5 of its 10 solutions, there is a 91.7% chance that at least one is correct. This gap between pass@1 and pass@k for larger $k$ is typical: models often know the solution well enough to generate it sometimes, but not consistently enough to make every attempt reliable.

The ratio between pass@1 and pass@k at larger values reveals something important about the model's confidence and consistency. A model that achieves pass@1 = 0.90 and pass@10 = 0.99 is being consistent: its solutions are almost always correct, and sampling more of them adds only marginal value. A model that achieves pass@1 = 0.40 and pass@10 = 0.99 is diverse but inconsistent: it clearly knows how to solve problems because it can generate correct solutions, but it cannot do so reliably in a single attempt. These are very different behavioral profiles with different practical implications, and reporting pass@k at multiple values of $k$ lets you distinguish them in a way that a single number cannot.

The pass@k metric rewards diversity. If all $k$ samples are essentially the same solution with the same bug, they all fail together and pass@k is no better than pass@1. If the samples explore different algorithmic strategies, variable naming choices, and control flow patterns, at least one approach might succeed even if others fail. Diversity is what makes pass@k larger than pass@1.

This creates a meaningful tension in how to sample. Greedy decoding (temperature $T = 0$) produces the single most likely output and is appropriate for pass@1 measurements where you want the model's single best guess. Higher temperatures ($T = 0.6$ to $0.8$) produce more diverse outputs and are appropriate for pass@k measurements where $k > 1$, because diversity increases the chances that at least one sample is correct.

Most benchmark evaluations use temperature 0.8 for measuring pass@k with $k > 1$, and either temperature 0 or 0.2 for pass@1.

There is a deeper tradeoff here. High temperature improves pass@k for large $k$ but degrades pass@1, because increased randomness means the model's most reliable solution is less likely to be generated on any given attempt. Low temperature improves pass@1 but can harm pass@k, because samples cluster around the same output and diversity collapses. Finding the temperature that maximizes a specific pass@k value for a specific deployment scenario is itself an optimization problem. Production code assistant designers often tune temperature separately for interactive completion (where pass@1 matters most) and background generation pipelines (where pass@k for moderate $k$ is the goal).