HumanEval › Temperature, Diversity, and the pass@k Tradeoff

One of the most practically important aspects of the pass@k evaluation is the interaction between temperature and performance. Temperature controls the randomness of the language model's sampling distribution: at temperature 0, the model always produces its single most likely next token (greedy decoding), creating deterministic output. At higher temperatures, the distribution becomes more uniform, increasing diversity at the potential cost of individual sample quality.

For pass@1, lower temperatures typically perform better. Greedy decoding or low-temperature sampling concentrates probability mass on the most likely sequence, which for well-trained models tends to be the correct solution when one exists. For pass@k with larger $k$, higher temperatures often improve performance. The intuition is that you want diverse samples: if the model generates the same wrong answer 100 times at temperature 0, pass@100 equals pass@1. Higher temperatures explore more of the probability space, increasing the chance that at least one sample is correct.

This tension creates a practical challenge: the optimal temperature for single-shot deployment differs from the optimal temperature for multi-sample evaluation. Research using the HumanEval benchmark found that models often benefit from separate temperature settings depending on the deployment context. Production coding assistants like GitHub Copilot typically operate at low temperatures to maximize pass@1 for immediate suggestions, while automated code repair systems can afford higher temperatures to explore more diverse solutions.

The original Codex paper used temperature 0.8 for sampling, which was found to give a good balance between diversity (needed for high $k$ estimates) and quality (needed for reasonable $c/n$ base rates). Subsequent work explored temperature annealing: starting with a high temperature for the first few samples to explore diverse approaches, then reducing temperature as the number of remaining samples decreases. This strategy is analogous to simulated annealing in optimization and can improve pass@k compared to fixed-temperature sampling.