Code Evaluation › Computing pass@k in Practice
Language AI HandbookAll chapters Code Evaluation › Computing pass@k in PracticeOne section · Sections in this chapter6 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let us implement the combinatorial pass@k estimator and apply it to a synthetic evaluation scenario. This will make the formula concrete and demonstrate the efficiency of computing multiple values from a single pool of samples.
from math import comb
def pass_at_k(n: int, c: int, k: int) -> float:
"""
Compute the unbiased pass@k estimate.
Parameters
----------
n : total number of samples generated per problem
c : number of correct samples among the n samples
k : number of samples shown or tried
Returns
-------
Estimated probability that at least one of k samples is correct.
"""
if n - c < k:
return 1.0
return 1.0 - comb(n - c, k) / comb(n, k)pass@ 1: 0.3000 pass@ 2: 0.5333 pass@ 5: 0.9167 pass@10: 1.0000
With total samples and correct ones, pass@1 is 0.30 and pass@5 is 0.917. Adding more diversity by trying 10 samples guarantees at least one correct solution whenever there are any correct solutions at all, since if and , the probability that all chosen solutions are wrong requires choosing all of them from the incorrect ones, which is impossible when there are fewer incorrect samples than the budget.
Estimating pass@k Across a Benchmark
In practice, you estimate pass@k for an entire benchmark by averaging the per-problem estimates:
from typing import List
import numpy as np
def benchmark_pass_at_k(problem_results: List[dict], k: int) -> float:
"""
Compute pass@k averaged over all problems in a benchmark.
Each element of problem_results should have keys 'n' and 'c'.
"""
per_problem = [pass_at_k(r["n"], r["c"], k) for r in problem_results]
return float(np.mean(per_problem))import random
random.seed(42)
np.random.seed(42)
# Simulate a benchmark with 164 problems (HumanEval scale)
# Model solves about 65% of problems when it tries many times (ceiling)
# but with realistic variance per problem
num_problems = 164
n_samples = 20 # Generate 20 samples per problem
simulated_results = []
for _ in range(num_problems):
# True per-problem difficulty drawn from a beta distribution
# to capture the spectrum from trivially easy to extremely hard
true_prob = np.random.beta(a=1.5, b=1.5) * 0.9
c = np.random.binomial(n=n_samples, p=true_prob)
simulated_results.append({"n": n_samples, "c": c})Simulated benchmark: 164 problems, 20 samples each pass@ 1: 0.445 (44.5%) pass@ 5: 0.840 (84.0%) pass@10: 0.925 (92.5%) pass@20: 0.970 (97.0%)
The results show the expected pattern: pass@1 is the hardest floor (the model must be reliable on each attempt), and pass@k increases as grows because sampling diversity helps find correct solutions among the pool. The rate of increase depends on how many problems are in the "sometimes solved" category versus the "never solved" category.
Pass Rate Breakdown by Difficulty
A useful supplement to the overall pass@k score is a difficulty breakdown. Problems where (no correct samples at all) represent complete failures: the model has no ability to solve these problems regardless of how many samples you take. Problems where represent trivially easy cases that the model solves reliably on every attempt. Problems in between, where , show the model's reliability gradient: it knows how to solve these problems sometimes but not consistently.
def difficulty_breakdown(problem_results: List[dict]) -> dict:
"""Categorize problems by how often the model solves them."""
total = len(problem_results)
never_solved = sum(1 for r in problem_results if r["c"] == 0)
always_solved = sum(1 for r in problem_results if r["c"] == r["n"])
sometimes_solved = total - never_solved - always_solved
return {
"total": total,
"never_solved": never_solved,
"never_pct": never_solved / total,
"sometimes_solved": sometimes_solved,
"sometimes_pct": sometimes_solved / total,
"always_solved": always_solved,
"always_pct": always_solved / total,
}Difficulty breakdown (164 problems, 20 samples each): Never solved (c=0): 5 problems (3.0%) Sometimes (0 < c < n): 158 problems (96.3%) Always solved (c=n): 1 problems (0.6%)
The "sometimes solved" category is the most interesting from an engineering perspective: these are problems the model understands but cannot solve reliably. Increasing diversity through higher temperature or more samples helps here more than it helps with "never solved" problems. For model improvement efforts, the "never solved" category is the target for capability expansion (the model needs to learn something new), while the "sometimes solved" category is the target for reliability improvements (the model knows the answer but needs to surface it more consistently).
This breakdown also reveals whether a benchmark is well-calibrated for a given model. If nearly all problems fall in "always solved," the benchmark is too easy and provides no useful signal for distinguishing the model from a more capable one. If nearly all problems fall in "never solved," the benchmark is too hard and tells you only that the model cannot handle problems at this difficulty level. A good benchmark has substantial mass in the "sometimes solved" category, which is where the measurement of capability gradient lives.
The visualization below shows how pass@k changes as increases, using three simulated models with different capability profiles.
The Effect of Temperature on pass@1 vs pass@k
The plot below illustrates the temperature tradeoff: how increasing sampling temperature shifts the balance between reliability (pass@1) and capability (pass@10). Understanding this tradeoff is essential for configuring code generation systems for their specific deployment context.
