Code Evaluation › Functional Correctness

Functional correctness is the standard criterion for code evaluation: a program is correct if it produces the expected output for all valid inputs. The simplest way to measure this is unit tests. Given a function specification and a set of (input, expected output) pairs, you run the generated function on each input and check whether the output matches.

This binary pass/fail criterion is strict. A function that produces the right answer for 9 out of 10 test cases but crashes on the 10th is marked incorrect. This strictness is intentional: in real software, a function that fails on edge cases causes bugs that can be hard to diagnose and costly to fix. The fact that it works "most of the time" is often not the point. The function was called because it was needed, and if it fails on the specific input that arrives in production, the failure is total regardless of its historical success rate.

The strictness also has an important consequence for evaluation: a model's functional correctness score will always be a lower bound on its true capability. A model that generates a nearly correct solution, one with a subtle bug in an unusual edge case, gets zero credit on that problem even though it clearly understood the task. This is a real cost, but the alternative of awarding partial credit based on how many test cases pass creates its own problems, as we discuss in the Limitations section.

A single test suite paired with a natural language specification is called a problem in the benchmark literature. The model receives the specification, sometimes along with a few example input/output pairs for context, and must generate a function that passes a hidden test suite that it has not seen.

The key design choice in a problem is the test suite itself. Test cases need to cover the full range of inputs the function might reasonably encounter:

• Normal cases: Typical inputs that the function should handle without difficulty
• Edge cases: Boundary values like empty inputs, single elements, zero, or maximum representable values
• Error cases: Inputs that should trigger exceptions or error handling
• Performance cases: Large inputs that would expose quadratic time complexity or other scalability issues

Most public benchmarks provide a handful of test cases per problem, which is enough to distinguish correct from obviously wrong implementations but often not enough to catch subtle bugs in almost-correct implementations. The EvalPlus project, which we discuss in detail later, found that augmenting test suites from the typical 7-8 tests to hundreds of generated tests substantially changes the apparent pass rates of models. This demonstrates how much benchmark scores depend on test suite quality and how sensitive the numbers can be to measurement choices that are rarely reported prominently.

A well-designed benchmark should report how many test cases each problem has, how those test cases were generated, and whether they were manually reviewed for correctness. These details affect how much to trust the numbers but are often omitted from papers that simply report a final pass rate.

Running generated code requires a controlled execution environment. You cannot execute arbitrary generated code on your main system without risks: the code might access the filesystem, make network requests, spawn processes, consume unlimited resources, or enter infinite loops.

Practical evaluation systems use sandboxed execution with resource limits:

• Timeout limits: Kill any program that runs longer than a threshold, typically 1-10 seconds per test case. This catches infinite loops and prevents a single problem from consuming all evaluation time.
• Memory limits: Prevent programs from allocating excessive memory, which can crash the host system or interfere with other evaluations running in parallel.
• Process isolation: Use containers or virtual machines to prevent generated code from affecting the host system, other evaluation processes, or any sensitive data on the machine.
• Import restrictions: Block potentially dangerous imports like os, subprocess, or socket that could enable code to escape the sandbox environment.

The HumanEval benchmark, which we will examine shortly, uses a relatively simple approach: execute code in a subprocess with a timeout, catch any exceptions, and compare outputs. More recent and security-conscious benchmarks add tighter sandboxing using technologies like Docker containers, gVisor, or Firecracker microVMs.

Execution safety is a serious engineering challenge for large-scale evaluation. When you are running hundreds of thousands of model-generated programs to evaluate a model family, even a small fraction of adversarial or buggy programs that escape the sandbox can cause real damage. Production evaluation infrastructure at major AI labs typically uses full container isolation with fine-grained resource controls that go significantly beyond what academic benchmarks demonstrate. The academic benchmarks prioritize reproducibility and simplicity; production evaluation prioritizes safety.