HumanEval › Key Parameters and Experimental Choices

Several design choices materially affect the numbers you report when evaluating against HumanEval. Understanding these choices is needed for correctly interpreting results and making fair comparisons across papers.

The key parameters are:

• n (samples per problem): Total number of samples generated per problem. This determines the precision of the estimate and must be greater than or equal to k. In research settings, n is typically set to 200 to ensure stable estimates while keeping computational costs manageable. Setting n too low results in high variance estimates, particularly for hard problems where c is small.
• c (correct count): Number of correct samples that pass all test cases. This is the observed count of successful solutions from the n samples, determined by executing each generated sample against the test harness and counting successes.
• k (evaluation budget): Number of samples selected for evaluation, representing the number of attempts allowed to solve the problem. Common values are 1 (single-shot accuracy), 10 (moderate search budget), and 100 (extensive search).
• Temperature: The sampling temperature used to generate each of the n samples. It directly affects the diversity and quality tradeoff described above.

Understanding the relationship between these parameters is important for interpreting results. A model might show modest pass@1 but excellent pass@100, showing high diversity in generation but inconsistent single-sample quality. Conversely, high pass@1 but low relative improvement at higher k suggests the model produces similar, high-quality outputs consistently. These profiles correspond to different practical use cases: the high-diversity model suits automated selection systems, while the high-precision model suits interactive assistants where users see only one suggestion.

A common source of confusion in the literature involves the distinction between pass@1 as measured with temperature sampling versus greedy decoding. Some papers report pass@1 using temperature 0 (greedy), which measures the model's most confident prediction. Others use temperature 0.8 and the formula above with $n = 200$, $k = 1$. These can differ substantially. Greedy decoding generally yields higher pass@1 for well-calibrated models, since the mode of the distribution is typically the most likely correct answer. The formula-based estimate from temperature sampling can be lower because high temperature introduces more noise. When comparing results across papers, you must check which method each paper uses.