Code Evaluation › Summary

Code evaluation has evolved from approximate string-matching metrics to execution-based testing that directly measures functional correctness, and from single-metric benchmarks toward multi-dimensional evaluation suites that capture the full complexity of code quality.

The key ideas are:

• Functional correctness is the primary criterion: a function is correct if it passes all test cases, and execution-based evaluation is the right way to measure it. String similarity metrics like BLEU fail because semantically identical functions can look very different, and semantically different functions can look very similar.
• pass@k quantifies the probability that at least one of $k$ sampled solutions is correct. The combinatorial estimator avoids sampling variance by using $n \geq k$ samples and computing the result analytically using the complement rule.
• pass@1 vs. pass@k: pass@1 measures reliability (can the model solve this on the first try?), while pass@k measures capability (can the model solve this at all given multiple attempts?). Reporting both reveals whether a model is consistent or merely capable.
• Major benchmarks: HumanEval established pass@k as the standard and measures basic function-writing in Python. MBPP extends breadth with more varied, naturalistic specifications. APPS tests algorithmic problem-solving at competitive programming difficulty. SWE-bench tests real-world bug fixing in full repositories. MultiPL-E evaluates performance across 18 programming languages. LiveCodeBench avoids contamination through continuous problem updates.
• Benchmark limitations: Small test suites inflate pass rates. EvalPlus-style augmentation reveals the gap. Contamination can overstate generalization. Competitive programming problems underrepresent practical engineering tasks. The distribution shift between HumanEval and SWE-bench scores reveals that different benchmarks test different capabilities.
• Beyond correctness: Evaluation is expanding to cover code style and idiom, security vulnerabilities, execution efficiency, and human preference, because functional correctness is necessary but not sufficient for production-ready code. Each dimension requires different tools and reveals different gaps.

As we move into production systems, the evaluation metrics covered in this chapter become the basis for deployment decisions: how often to show model suggestions, when to require human review, and how to monitor model quality in live environments where ground truth is not immediately available. A model that solves 90% of HumanEval problems but only 25% of SWE-bench issues needs different deployment guardrails than one that performs well on both.