Code Evaluation › Why Code Evaluation Is Hard
Language AI HandbookAll chapters Code Evaluation › Why Code Evaluation Is HardOne section · Sections in this chapter2 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Evaluating code is harder than evaluating natural language text, and the difficulty comes from a fundamental asymmetry: there are many correct programs for any given specification, but there are also infinitely many incorrect ones. You cannot resolve this asymmetry by counting words or comparing strings.
For natural language generation, metrics like BLEU or ROUGE measure n-gram overlap between a generated string and a reference string. These metrics are imperfect proxies for quality, but they are at least sensible proxies when there is one canonical way to express an idea. Code is different. Two functions that use entirely different variable names, loop structures, and even algorithms can both be correct implementations of the same specification. A string-similarity metric will penalize both if they differ from the reference, even when both are functionally equivalent to the reference and to each other.
The alternative is execution-based evaluation: run the code, feed it test inputs, and check whether the outputs match. This sidesteps the equivalence problem completely. You do not need to compare structure or syntax. You just need to know whether the program behaves correctly on the cases you care about. This is the approach that dominates modern code evaluation, and it is the foundation on which everything else in this chapter builds.
Early work on code evaluation tried to get around the equivalence problem by asking human judges to rate code quality. This is accurate but slow and expensive. It also does not scale to the kind of continuous, automated evaluation that model development requires, where you might need to evaluate thousands of model outputs every time you adjust a training hyperparameter. Execution-based testing offers the best compromise: it is objective, automatable, and captures what matters about code from a user perspective. If the code runs correctly on the inputs you care about, that is what matters, regardless of how it looks.
The Test Coverage Problem
Execution-based evaluation is more principled than string matching, but it has its own weakness: you can only test what you can test. A function that passes every test case in your suite might still fail on inputs you did not think to include. This is the test coverage problem, and it is fundamental to the structure of software testing, not an accidental limitation of any particular benchmark.
Consider a function that is supposed to sort a list. If your test suite only checks non-empty lists of integers, a buggy implementation that crashes on empty input or misbehaves on lists containing duplicates will appear correct. The model's pass rate on your suite is not the same as its true correctness rate, because the suite is an incomplete specification of what "correct sorting" means.
This gap matters because it affects how you interpret evaluation results. A model scoring 80% on a benchmark does not mean 80% of its outputs are correct programs in any absolute sense. It means 80% of the benchmark's test inputs are handled correctly by those programs. The gap between benchmark performance and production reliability depends entirely on how representative the benchmark's test cases are of the inputs the code will encounter in deployment.
The test coverage problem is well-known in traditional software testing, and it has generated an entire field of research around test generation, mutation testing, and formal verification. For code evaluation benchmarks, the practical implication is that benchmark designers must invest seriously in test suites with broad input coverage, and researchers must be appropriately skeptical of benchmark numbers when test suites are thin. We will see this tension recur when we examine specific benchmarks later in this chapter. The EvalPlus project made this concrete by showing that augmenting HumanEval's test suites substantially changes apparent pass rates, not because models got better or worse, but because better tests revealed bugs the original tests missed.
Beyond Functional Correctness
Even if a program passes all tests, it might still be unsuitable for production use. Evaluation researchers have identified at least four dimensions of code quality beyond functional correctness:
- Style and readability: Does the code follow the language's conventions, use meaningful names, and communicate intent clearly?
- Efficiency: Is the algorithm efficient, or does it use a quadratic loop where a linear one would serve equally well?
- Security: Does the code introduce SQL injection vulnerabilities, buffer overflows, or insecure cryptographic choices?
- Maintainability: Is the code well-structured, properly documented, and easy to modify when requirements change?
Most current benchmarks focus almost exclusively on functional correctness because it is the easiest dimension to automate. The richer dimensions require either human judgment or specialized static analysis tools. This is not a permanent limitation but a reflection of where the field currently is. We will discuss what is currently possible along each dimension at the end of this chapter, and the trend is clearly toward evaluation across more dimensions as the tools mature.
Why Text Similarity Metrics Fail for Code
Before diving into execution-based evaluation in detail, it is worth understanding concretely why string-similarity approaches fail for code. This motivates the design choices that make execution-based benchmarks better and explains why the transition away from BLEU-style metrics was necessary.
Suppose a model generates this implementation of a function to find the maximum element in a list:
def find_max(lst):
result = lst[0]
for item in lst:
if item > result:
result = item
return resultA reference implementation might be:
def find_max(numbers):
maximum = numbers[0]
for num in numbers:
if num > maximum:
maximum = num
return maximumThese are functionally identical but share almost no tokens beyond structural keywords like def, for, if, and return. BLEU would give this a very low score, even though the generated code is correct and arguably just as readable as the reference. Meanwhile, a model that generates:
def find_max(lst):
# Find the maximum element in the list
return max(lst)would also score poorly on BLEU relative to the reference, despite being a more idiomatic and arguably better Python solution that any experienced Python developer would prefer. The comment and the use of the built-in max function are improvements over the explicit loop, but string similarity cannot see that.
Now consider a model that generates nearly identical code with a subtle off-by-one error:
def find_max(lst):
result = lst[1] # Bug: starts from index 1, not 0
for item in lst:
if item > result:
result = item
return resultThis version would score relatively high on BLEU because most tokens match the reference exactly, even though the function is incorrect for single-element lists (it raises an IndexError) and for lists where the maximum element is at index 0. Execution-based testing immediately catches this bug; BLEU rewards the match and cannot see the error.
This asymmetry, rewarding look-alike incorrect code and penalizing functionally equivalent correct code, is the fundamental problem with text similarity metrics for code. Execution-based evaluation resolves both issues simultaneously.