MBPP › Key Parameters for Evaluation

Designing an MBPP evaluation run requires choosing values for several parameters that materially affect the results:

• $n$ (samples per problem): The total number of candidate solutions to generate. The original paper used $n = 200$, which gives stable estimates but requires substantial computation. Smaller values like $n = 20$ or $n = 50$ are often used when evaluating large models with expensive inference. Smaller $n$ increases variance in the estimates but is often acceptable for comparing models that differ by several percentage points.
• $c$ (correct count): This is determined empirically by running all $n$ samples against the test suite. It is not a hyperparameter but rather an observed quantity per problem.
• $k$ (samples to select): The published metric. Common choices are $k = 1$ (single-shot performance), $k = 10$ (performance with limited retries), and $k = 100$ (upper bound on performance given many attempts). Most modern papers report pass@1 as the primary metric since it best shows real-world usage scenarios.
• Timeout: The maximum execution time per candidate. MBPP problems are computationally lightweight, so a 5-second timeout is usually sufficient. Longer timeouts increase evaluation time; shorter timeouts risk false negatives for correct solutions that happen to be computationally inefficient.
• Temperature: The sampling temperature used during generation. Higher temperatures produce more diverse samples, which can increase pass@k for larger $k$ but may decrease pass@1. Temperature 0 (greedy decoding) is often used for pass@1 evaluation.