Code Evaluation › Key Evaluation Parameters

Designing a code evaluation run well requires careful choices about several parameters that interact in non-obvious ways. Getting these right matters for obtaining reliable estimates and for making results comparable across papers.

The key parameters for code evaluation are:

• $n$ (samples per problem): More samples give a more stable estimate but cost proportionally more compute. $n = 200$ is common for careful research evaluations. For rapid iteration during model development, $n = 20$ or $n = 50$ is more tractable while still providing useful directional signal.
• $k$ (target budget): The deployment scenario determines which $k$ values matter. Pass@1 is the right metric for user-facing tools where the model must produce a correct answer immediately. Pass@10 or pass@100 is appropriate for pipeline systems where multiple candidates are generated and filtered by automated tests.
• Temperature: Lower temperatures for pass@1 measurements (exploit the model's best guess consistently), higher temperatures for pass@k with $k > 1$ (explore diverse solutions that increase the odds of at least one being correct).
• Test suite size: Larger test suites catch more bugs and give more trustworthy numbers. EvalPlus-style augmentation reduces the gap between benchmark performance and real-world correctness. When comparing models, prefer benchmarks that report test suite size per problem.
• Benchmark freshness: Older benchmarks are more likely to be contaminated by training data. LiveCodeBench and SWE-bench's reliance on post-cutoff content makes them more robust to contamination than static benchmarks built from problems that circulated online before model training.