Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

HumanEval › Code Implementation

Michael BrenndoerferMarch 2, 20269 min read

Language AI HandbookAll chapters

Part 54 of 67Benchmark Evaluation

Chapter 418HumanEval › Code Implementation

Reading section 8 of 15 · Code Implementation · 9 minSection 8 of 15 · 9 min

Section 8 of 15 · 9 min read

HumanEval › Code ImplementationOne section · Sections in this chapter8 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let us implement the pass@k calculation and examine the HumanEval dataset structure. We will use the math library for exact combinatorial calculations and implement the metric from scratch to ensure understanding.

In[4]:

Code

import math

# Mock dataset for demonstration: 164 problems, each with required fields
heval = [
    {
        "task_id": f"HumanEval/{i}",
        "prompt": 'def example():\n    """Example function."""\n    pass\n',
        "canonical_solution": "    return True",
        "test": "def check(func):\n    assert func() is True\n",
        "entry_point": "example",
    }
    for i in range(164)
]


def calculate_pass_at_k(n, c, k):
    """
    Calculate pass@k given n total samples and c correct samples.

    Uses the exact hypergeometric formula:
        pass@k = 1 - C(n-c, k) / C(n, k)

    Args:
        n: Total number of samples generated per problem
        c: Number of correct samples (passing all tests)
        k: The k in pass@k

    Returns:
        float: pass@k score between 0 and 1
    """
    if n < k:
        raise ValueError(f"n ({n}) must be >= k ({k})")
    if c > n:
        raise ValueError(f"c ({c}) cannot be > n ({n})")
    if c == 0:
        return 0.0
    if c == n:
        return 1.0
    # When n - c < k, all k selections must include at least one correct
    if n - c < k:
        return 1.0
    return 1.0 - math.comb(n - c, k) / math.comb(n, k)


def pass_at_k_approximation(n, c, k):
    """
    Approximate pass@k using the independence assumption.
    Valid when n >> k.

    pass@k ≈ 1 - (1 - c/n)^k
    """
    if n == 0:
        return 0.0
    return 1.0 - (1 - c / n) ** k

Out[5]:

Console

HumanEval contains 164 problems

Keys in each example: ['task_id', 'prompt', 'canonical_solution', 'test', 'entry_point']

The dataset contains 164 programming problems. Let us examine the structure of a specific example.

In[6]:

Code

# Examine the first problem
example = heval[0]

task_id = example["task_id"]
prompt = example["prompt"]
canonical = example["canonical_solution"]
test_code = example["test"]
entry_point = example["entry_point"]

Out[7]:

Console

Task ID: HumanEval/0
Entry point (function to implement): example

--- PROMPT ---
def example():
    """Example function."""
    pass


--- CANONICAL SOLUTION ---
    return True

--- TEST CODE ---
def check(func):
    assert func() is True

The prompt gives the function signature and docstring describing what the function should do. The canonical solution shows one correct implementation, and the test code contains assertions that any correct implementation must satisfy. A model under evaluation receives only the prompt and must generate the body of the function.

Now let us verify the metric calculation matches our worked example.

Out[8]:

Console

Exact pass@k calculations for n=5, c=2:
  pass@1: exact=0.400  approx=0.400
  pass@2: exact=0.700  approx=0.640
  pass@3: exact=0.900  approx=0.784
  pass@4: exact=1.000  approx=0.870
  pass@5: exact=1.000  approx=0.922

Boundary condition checks:
  pass@5 with c=2 (guaranteed hit all in 5 draws): 1.000
  pass@1 with c=0 (no correct samples): 0.000
  pass@1 with c=5 (all samples correct): 1.000

Out[9]:

Visualization

Bar chart showing pass@k values for k from 1 to 5 with n=5 samples and c=2 correct. Bars increase from 0.4 at k=1 to 1.0 at k=5. — Pass@k values for the worked example with n=5 and c=2. The probability of finding at least one correct solution increases monotonically from 0.4 at k=1 to 1.0 at k=5, which shows the hypergeometric distribution over the pool of samples.

Out[10]:

Visualization

Line chart comparing exact and approximate pass@k for n=20 and k=10 with success rate on x-axis. — Exact versus approximate pass@k for n=20 and k=10. The approximation shows slight deviation from the exact formula at small n, particularly at intermediate success rates, because the independence assumption becomes less accurate when k is a larger fraction of n.

Line chart comparing exact and approximate pass@k for n=100 and k=10. Lines closely overlap. — Exact versus approximate pass@k for n=100 and k=10. The two calculations track each other closely across the full range of success rates, validating the approximation for typical experimental settings.

Line chart comparing exact and approximate pass@k for n=200 and k=10. Lines are nearly identical. — Exact versus approximate pass@k for n=200 and k=10. At this larger n-to-k ratio, the approximation becomes nearly indistinguishable from the exact hypergeometric calculation, confirming asymptotic convergence.

The calculations match our manual worked example. Notice how the approximation diverges slightly as $k$ approaches $n$ , which is expected since the approximation assumes $n \gg k$ . The exact calculation properly handles the saturation at pass@k = 1 when $k$ exceeds the number of incorrect samples.

Simulating a Multi-Problem Benchmark EvaluationLink Copied

Now let us show how pass@k is aggregated across multiple problems with varying difficulty. We will simulate generating $n = 20$ samples per problem with varying underlying success rates.

In[11]:

Code

num_problems = 10
n_samples = 20  # Number of samples generated per problem

# Simulate c (number correct) for each problem
# Beta distribution models varying difficulty: Beta(2, 5) skews toward low success rates
underlying_rates = np.random.beta(2, 5, num_problems)
c_vals_sim = np.random.binomial(n_samples, p=underlying_rates)

problem_results = [
    {
        "problem_id": f"problem_{i}",
        "n": n_samples,
        "c": int(c_vals_sim[i]),
        "true_rate": float(underlying_rates[i]),
    }
    for i in range(num_problems)
]

Out[12]:

Console

Simulated evaluation results:
Problem ID   | n         | c           | pass@1 | pass@5 | pass@10
----------------------------------------------------------------------
problem_0    |        20 |           6 |   0.30 |   0.87 |    0.99
problem_1    |        20 |           6 |   0.30 |   0.87 |    0.99
problem_2    |        20 |           7 |   0.35 |   0.92 |    1.00
problem_3    |        20 |           3 |   0.15 |   0.60 |    0.89
problem_4    |        20 |          11 |   0.55 |   0.99 |    1.00
problem_5    |        20 |           1 |   0.05 |   0.25 |    0.50
problem_6    |        20 |           6 |   0.30 |   0.87 |    0.99
problem_7    |        20 |           5 |   0.25 |   0.81 |    0.98
problem_8    |        20 |           7 |   0.35 |   0.92 |    1.00
problem_9    |        20 |          10 |   0.50 |   0.98 |    1.00

Out[13]:

Visualization

Grouped bar chart comparing pass@1, pass@5, and pass@10 for 10 simulated problems. Problems vary in difficulty and pass@10 consistently exceeds pass@1. — Pass@k scores across 10 simulated problems of varying difficulty (n=20 samples each). Every problem shows improvement as k increases, but problems with more correct samples see larger absolute gains. The clustering of problems at 0.0 for pass@1 highlights how difficult problems may yield no correct samples in small batches.

The simulated results reveal large variation in problem difficulty, with easier problems showing high pass rates even at k=1 while harder problems benefit materially from increased sampling budgets. Notice how pass@k consistently increases with k for every problem. This shows the value of generating multiple candidate solutions. This pattern mirrors practical development workflows where we iterate through several attempts before finding a working solution. The diminishing returns become apparent as well: increasing k from 1 to 5 often yields large improvements, while increasing from 5 to 10 gives smaller gains for problems with few correct samples.

Aggregating Across the BenchmarkLink Copied

Finally, we aggregate across all problems to get the overall benchmark score. This is the number reported in papers as the model's pass@k performance on HumanEval.

In[14]:

Code

from collections import defaultdict

import numpy as np


def evaluate_benchmark(problem_results, k_values=[1, 10, 100]):
    """
    Calculate aggregate pass@k across all problems.
    """
    scores = defaultdict(list)

    for result in problem_results:
        n_r = result["n"]
        c_r = result["c"]
        for k in k_values:
            if k <= n_r:
                p = calculate_pass_at_k(n_r, c_r, k)
                scores[k].append(p)

    # Macro-average across problems
    aggregated = {k: np.mean(scores[k]) for k in scores}
    return aggregated

Out[15]:

Console

Aggregate pass@k scores across benchmark:
  pass@1: 0.310
  pass@5: 0.808
  pass@10: 0.936
  pass@20: 1.000

Out[16]:

Visualization

Line chart showing aggregate pass@k score as k increases from 1 to 20. The curve rises steeply early and flattens toward a plateau. — Aggregate pass@k across 10 simulated problems as k increases from 1 to 20, showing the characteristic improvement curve. The steep initial rise shows high marginal returns from the first few additional samples, while the curve flattens as problems with zero correct samples in the pool cannot contribute additional gains.

The aggregate curve shows how letting multiple attempts substantially increases the effective capability of the model. With a pass@1 of approximately 28%, the model solves roughly a quarter of problems on the first attempt, but increasing the sample budget to 10 attempts can raise performance substantially. This shows how generation diversity compensates for individual sample uncertainty. This exponential improvement curve helps practitioners decide how many samples to generate based on their latency and compute budgets versus accuracy requirements.