Code Evaluation › Beyond Correctness

Functional correctness is necessary but not sufficient for high-quality code. Evaluation researchers are extending the criteria in several directions, motivated by the observation that correctness alone does not predict whether code will be useful, maintainable, or safe in production.

A model might generate code that passes all tests but is poorly structured, hard to read, or contrary to the language's idioms. Several approaches to measuring code quality exist, and they capture different dimensions of the quality problem.

Static analysis tools like pylint and flake8 for Python, or ESLint for JavaScript, flag style violations, unused variables, overly complex functions, and other maintainability issues. Running these tools on model outputs gives a rough quality signal without human annotation. These tools were developed for human-written code, and they translate reasonably well to evaluating generated code, though generated code can sometimes trip their heuristics in unexpected ways. A model that generates highly nested code to avoid using a library function might trigger a "function too complex" warning even if the code is technically correct and readable.

Cyclomatic complexity measures how many independent execution paths exist through a function. A function with high cyclomatic complexity is hard to test exhaustively and difficult to maintain when requirements change. Models sometimes generate solutions with unnecessary nesting or redundant conditional branches where a cleaner boolean expression or an early return would suffice. This happens partly because training data includes a wide range of coding styles, some of which favor explicit branching over compact expressions.

Idiomatic code scoring compares model outputs against the distribution of real code in the language's standard libraries and popular open-source projects. A Python function that reimplements the behavior of itertools.groupby from scratch rather than importing it is technically correct but idiomatically poor, and any experienced Python developer would find it surprising. This approach requires a reference corpus and a way to compute distributional similarity, but it captures an important quality dimension that binary pass/fail testing misses entirely. Idiomatic code is easier to read, easier to maintain, and benefits from the testing and optimization that went into the standard library implementation.

Security bugs in generated code are a real and growing concern. Studies have shown that Copilot and similar tools can generate code containing CWE (Common Weakness Enumeration) vulnerabilities, including SQL injection through string interpolation, path traversal through unvalidated user input, and use of deprecated cryptographic functions with known weaknesses.

The SecurityEval benchmark tests models specifically on security-sensitive programming scenarios, asking them to generate code for tasks like reading files, making database queries, handling authentication, or processing user input. Each problem has ground truth labels for secure versus insecure patterns, allowing automated scoring of whether the generated code avoids known vulnerability classes.

Evaluating security thoroughly requires domain expertise and cannot be fully automated with current tools. Static analyzers can catch some vulnerability patterns, but sophisticated security issues often require understanding the deployment context rather than just the code itself. A function that reads a file whose path comes from user input might look fine in isolation but creates a directory traversal vulnerability when deployed in a web server context. Evaluating this correctly requires knowing the deployment context, which automated tools typically cannot infer from the function alone. This contextual gap means that security evaluation is necessarily incomplete for any purely automated benchmark.

Two functions can both be functionally correct but differ by orders of magnitude in runtime or memory usage. A bubble sort and a mergesort both sort lists correctly, but on 10,000 elements, the difference is hundreds of milliseconds versus a few milliseconds. On 1,000,000 elements, one function is still fast and the other never returns within any reasonable time limit.

Efficiency evaluation typically works by first verifying that a solution is correct, then measuring its runtime on large inputs and comparing against the fastest known solution for each problem. APPS competition problems naturally test algorithmic efficiency because the judge's time limits are set to require efficient implementations. A solution that passes the correctness check but times out on large inputs is counted as incorrect in the competition context, which directly incentivizes efficient algorithms.

For models deployed in performance-sensitive contexts, including embedded systems, high-frequency trading, and real-time data processing, efficiency is a critical evaluation criterion. A model that generates provably correct but quadratic solutions to problems that admit linear ones is providing less value, even if it scores well on correctness benchmarks. The efficiency dimension is underrepresented in current evaluation frameworks but represents a real gap between what benchmarks measure and what deployed systems require.

The visualization below illustrates why efficiency matters beyond correctness, showing how the runtime gap between algorithmic complexity classes grows dramatically with input size.

Beyond automated metrics, human evaluators can assess qualities that are hard to formalize: whether code is readable, whether the solution approach is clever or unnecessarily convoluted, whether comments are helpful or misleading, and whether the variable names communicate intent. Human evaluation is expensive and slow to scale, but it is often more predictive of real-world usefulness than automated metrics alone, because it captures overall quality in the way a working developer would experience it.

Some researchers have used pairwise preference studies: show two generated solutions to a developer, ask which they prefer, and aggregate preferences using Elo-style rankings. This captures overall quality without requiring evaluators to specify exactly what they are measuring. Human preference studies have revealed that developers weight code readability and clarity more heavily than automated metrics suggest, which points toward a gap between what benchmarks measure and what developers value when using code assistants in their daily work.

An important finding from human preference studies is that correctness and preference do not always align. Developers sometimes prefer elegant incorrect code over ugly correct code, at least until they run the tests. They also prefer solutions that match the idioms of the codebase they are working in, which is something that automated metrics cannot assess without access to the surrounding codebase context. This suggests that preference-based evaluation captures different information than pass-rate benchmarks and complements rather than duplicates them.

The most ambitious evaluations combine multiple criteria to capture the full complexity of code quality. CRUXEval (Code Reasoning Understanding and eXecution Evaluation) tests whether models can reason about code execution in both directions: given a function and an output, predict the input that produced it; or given a function and an input, predict the output. This tests understanding of code semantics rather than just generation ability.

CodeReviewer asks models to review existing code and identify bugs, then evaluates whether the identified bugs are real and the suggested fixes are correct. This tests a different capability than generation: the ability to critically analyze code written by someone else and provide useful feedback. Many professional developers spend as much time reviewing code as writing it, so this is a practically important capability that pure generation benchmarks miss.

The trend in evaluation is clear and consistent: the field is moving toward multi-dimensional evaluation suites that capture the full complexity of software engineering, rather than single-number benchmarks that reduce quality to a pass rate on function-writing exercises. This is the right direction conceptually, but it comes with coordination costs. As evaluation becomes more multidimensional, comparing models across papers becomes harder because different papers use different combinations of metrics, weight them differently, and make different choices about benchmark design. The field will need to develop conventions for multi-dimensional reporting similar to the conventions that developed around HumanEval pass@1.