HumanEval › Functional Correctness vs. Lexical Similarity

Before HumanEval, code generation models were often evaluated using BLEU scores against reference implementations. BLEU (Bilingual Evaluation Understudy) counts matching n-grams between generated and reference text, awarding higher scores for longer matching sequences. However, code exhibits a many-to-one mapping between syntax and semantics: numerous syntactically different implementations can solve the same problem correctly.

Consider a function that computes the sum of a list. A reference implementation might use a for loop with an accumulator variable, while a generated solution uses Python's built-in sum() function, or perhaps numpy.sum(), or a recursive approach, or a list comprehension with sum(). Lexically, these share few n-grams. The loop-based solution contains tokens like for, in, range, and +=, while the built-in solution contains just return sum(nums). A BLEU score would penalize the correct built-in solution heavily, despite it being more Pythonic and efficient. Conversely, a buggy implementation might differ by only a single character (changing + to - or < to <=), achieving high lexical similarity while being completely wrong or failing on edge cases.

HumanEval discards lexical comparison entirely. A generated solution is correct if and only if it passes all test cases when executed. This binary judgment, pass or fail, eliminates the ambiguity of partial credit. It also aligns evaluation with the ultimate purpose of code generation: creating runnable, correct programs that satisfy specifications. This execution-based approach captures semantic equivalence regardless of syntactic form, recognizing that there are infinitely many ways to write correct code.

This approach requires executing untrusted code, necessitating security measures like sandboxing, timeout limits, and resource constraints. Generated code could theoretically contain infinite loops, attempts to delete files, or memory exhaustion attacks. The evaluation harness runs each generated solution in an isolated environment, catching exceptions, timeouts, and infinite loops as failures. This sandboxing adds computational overhead compared to string matching, but gives definitive evidence of correctness.

The sandboxing problem is non-trivial in practice. A naive execution environment allows generated code to import arbitrary libraries, make network requests, read environment variables containing API keys, or spawn subprocesses. Production evaluation frameworks like the human-eval package released by OpenAI use Unix process isolation, memory limits, and execution timeouts to constrain these risks. The timeout limit also handles infinite loops: if a solution does not terminate within a fixed time (typically a few seconds per test case), it is marked as failing. This prevents pathological cases where a model generates an exponential-time algorithm that technically produces the correct answer for small inputs but times out on larger test cases.

The difficulty of secure code execution has prevented some researchers from running HumanEval evaluations in-house, instead relying on cloud-based evaluation services. This creates a reproducibility challenge: different sandboxing implementations may produce slightly different results, and slight differences in Python version or library availability can cause edge-case test failures. The field has gradually converged on standardized evaluation containers to address this, but it remains a source of subtle inconsistency in reported numbers.