MBPP › Execution-Based Evaluation

Both MBPP and HumanEval rely on execution-based evaluation rather than textual similarity metrics like BLEU or ROUGE. Traditional NLP metrics fail for code generation because syntactically different solutions can be semantically identical, while similar-looking code can behave completely differently.

The basic issue is that code is not text to be compared, but instructions to be executed. Two solutions might look completely different, one using list comprehensions and another using explicit loops, yet produce identical outputs for all inputs. Conversely, two code snippets might differ by only a single character, perhaps a less-than versus less-than-or-equal sign, yet produce drastically different behaviors. Execution-based evaluation cuts through this surface variation to measure the relevant behavior: does the code do what it is supposed to do?

This principle sounds obvious but imposes concrete requirements on evaluation infrastructure. Running generated code requires a test harness that can execute Python in a controlled environment, catch exceptions, enforce time limits, and aggregate results across potentially hundreds of candidates per problem. Building this infrastructure correctly involves solving problems in software engineering, security, and systems programming that have nothing to do with natural language processing.

The standard evaluation metric for MBPP is pass@k, which measures the probability that at least one out of $k$ generated samples passes all test cases. This metric acknowledges the stochastic nature of language model generation: even capable models might produce incorrect code on any single attempt due to sampling randomness. By evaluating multiple samples and asking whether at least one succeeds, pass@k gives a more complete picture of model capability.

The intuition behind pass@k is straightforward. If you can only see one output from the model (the case $k = 1$), pass@1 tells you the probability that this single output is correct. This is the most relevant metric for applications where you show users a single suggestion. If you can generate multiple candidates and use a reranker, an execution sandbox, or other selection mechanism to pick the best one (the case $k > 1$), then pass@k for larger $k$ tells you the probability that at least one acceptable solution exists among the candidates.

To compute pass@k rigorously, we treat the selection of $k$ samples from $n$ total candidates as a hypergeometric sampling problem. Suppose we generate $n$ candidate solutions for a problem and find that $c$ of them are correct (pass all tests). We want the probability that a random selection of $k$ of these candidates contains at least one correct solution. The complement approach yields:

$$\text{pass@k} = \mathbb{E}_{\text{Problems}}\left[1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}\right]$$

where:

• $n$: the total number of samples generated per problem (typically 200 in the original paper), representing the complete pool of candidate solutions from which we draw
• $c$: the number of correct samples, meaning those that pass all test assertions, representing the successful candidates within the pool
• $k$: the number of samples we select for evaluation (typically 1, 10, or 100), representing how many attempts the model gets before we check for success
• $\binom{n-c}{k}$: the number of ways to choose $k$ items exclusively from the $n-c$ incorrect candidates, which is all selections containing no correct solution
• $\binom{n}{k}$: the total number of ways to choose any $k$ samples from all $n$ candidates
• $\mathbb{E}_{\text{Problems}}$: the expectation computed by averaging across all problems in the evaluation set

This formula works by computing the complement of the probability of failure. The fraction $\binom{n-c}{k} / \binom{n}{k}$ is the hypergeometric probability of drawing zero successes when selecting $k$ items from a population of $n$ items containing $c$ successes. Subtracting this from 1 yields the probability of obtaining at least one correct sample.

When $n$ is large relative to $k$, the hypergeometric distribution approximates a binomial distribution. In this regime, each draw is approximately independent with success probability $p = c/n$. The probability of failing all $k$ independent draws is then $(1 - p)^k$, giving:

$$\text{pass@k} \approx 1 - \left(1 - \frac{c}{n}\right)^k$$

where:

• $\frac{c}{n}$: the empirical probability that any single randomly generated sample is correct, estimated from the pool of $n$ samples
• $\left(1 - \frac{c}{n}\right)^k$: the probability under independent Bernoulli sampling that all $k$ draws are incorrect

For the specific case of $k = 1$, this simplifies to:

$$\text{pass@1} \approx \frac{c}{n}$$

which is simply the fraction of correct samples in the pool. This makes intuitive sense: if 20 out of 100 generated samples are correct, the probability that a randomly chosen single sample is correct is 20%.

For the case where the per-sample success probability is small, we can use the first-order Taylor expansion of the binomial to obtain an even simpler approximation. When $p = c/n \ll 1$:

$$\text{pass@k} \approx 1 - (1-p)^k \approx 1 - e^{-kp} \approx kp = \frac{k \cdot c}{n}$$

This linear approximation holds when $kp$ is small, giving the intuitive result that pass@k scales roughly linearly with both $k$ and the base success probability. However, this approximation breaks down as $k$ grows, since pass@k is bounded above by 1 but the linear approximation is not.

The key reason for using the exact combinatorial formula rather than the approximation is numerical stability and unbiasedness. The approximation introduces upward bias when $c$ is small relative to $n$, precisely the regime where accurate estimation matters most for identifying the capability ceiling of weak models. The exact formula is unbiased regardless of the relationship between $c$ and $n$.

Let us work through a concrete example to build intuition. Suppose we generate $n = 20$ candidate solutions for a problem and find that $c = 4$ of them pass all test cases. We want to compute pass@k for $k \in \{1, 5, 10\}$.

For $k = 1$:

$$\begin{aligned} \text{pass@1} &= 1 - \frac{\binom{20-4}{1}}{\binom{20}{1}} \\ &= 1 - \frac{\binom{16}{1}}{\binom{20}{1}} \\ &= 1 - \frac{16}{20} \\ &= 0.200 \end{aligned}$$

With 4 correct solutions out of 20, we have a 20% chance that any single sample is correct. This equals $c/n$ exactly, confirming the simplified formula for $k = 1$.

For $k = 5$:

$$\begin{aligned} \text{pass@5} &= 1 - \frac{\binom{16}{5}}{\binom{20}{5}} \\ &= 1 - \frac{4368}{15504} \\ &= 1 - 0.2817 \\ &= 0.7183 \end{aligned}$$

With 5 samples, we have a 71.8% chance of including at least one correct solution. The approximation gives $1 - (1 - 0.2)^5 = 1 - 0.8^5 = 1 - 0.328 = 0.672$, which is noticeably lower due to the finite population correction. Drawing without replacement makes it easier to get diverse samples, which increases the chance of hitting a correct one.

For $k = 10$:

$$\begin{aligned} \text{pass@10} &= 1 - \frac{\binom{16}{10}}{\binom{20}{10}} \\ &= 1 - \frac{8008}{184756} \\ &= 1 - 0.0433 \\ &= 0.9567 \end{aligned}$$

With 10 samples drawn from our pool of 20 (which contains 4 correct solutions), we have a 95.7% chance of selecting at least one correct solution. The large improvement from 20% at $k=1$ to 95.7% at $k=10$ illustrates why the choice of $k$ matters so much for comparing models. A model that generates correct code 20% of the time at $k=1$ is not useless: if given 10 attempts and a way to identify the correct solution, it succeeds 96% of the time.

Calculating pass@k requires executing generated code against test cases in a controlled environment. This introduces several engineering challenges that do not exist in text-based evaluation.