Code Evaluation › Limitations of Current Evaluation

Even the best current evaluation frameworks have significant gaps that the research community is actively working to address. Understanding these gaps helps you interpret benchmark numbers more accurately and design evaluation strategies appropriate for specific deployment contexts.

Benchmarks are built at a point in time, and both models and benchmarks evolve in ways that create distribution shift. A benchmark that was challenging when introduced becomes saturated quickly as models improve, and the community then needs to build harder successors. But harder benchmarks often move toward competitive programming challenges that test algorithmic puzzle-solving rather than practical software engineering, which widens the gap between benchmark performance and the tasks developers face.

The broader distribution shift issue is between benchmarks and real code in production. Developers do not spend most of their time implementing sorting algorithms or string manipulation functions. They write database queries, API integrations, infrastructure configuration, authentication code, test code, and documentation. They debug issues in systems with complex state and interleaved concerns. Most benchmarks significantly underrepresent these practical tasks, not because benchmark designers do not understand this gap but because these tasks are much harder to evaluate automatically.

A model that scores 90% on HumanEval and 5% on SWE-bench is exhibiting exactly this distribution shift problem: excellent at toy functions, much weaker at real-world engineering tasks. The gap is not a failure of evaluation methodology but a real difference in capability that different benchmarks reveal. Reporting only HumanEval numbers gives a misleading picture of model usefulness for actual software development work.

The visualization below shows the performance gap between function-level benchmarks and repository-level benchmarks for hypothetical model families, illustrating that benchmark score does not transfer linearly across task types.

Natural language specifications are inherently ambiguous, and this creates a category of evaluation errors that are not the model's fault. A docstring that says "sort the list" leaves open questions: ascending or descending? In place or return a new list? Stable sort or not? What should happen for inputs containing non-comparable types? Models that generate code satisfying one reasonable interpretation may fail tests written for another equally reasonable interpretation.

More careful benchmarks try to write unambiguous specifications, but ambiguity is difficult to fully eliminate from natural language. This is also realistic in an important sense: real programming specifications written by product managers, issue trackers, and coworkers are often underspecified, and a good code assistant should either ask for clarification or generate code that covers the most common interpretation and documents its assumptions. Some recent evaluation work has started studying ambiguity resolution explicitly, asking models to identify underspecified requirements and generate clarifying questions before writing code. This capability is arguably more valuable than the ability to implement clearly specified functions, yet it is almost entirely absent from current benchmarks.

If a model's training data contains the solutions to benchmark problems, or even the problem statements themselves, its performance overstates its true generalization ability. This is a serious concern for benchmarks built from problems that appear on public coding websites, because competitive programming problems and their solutions are almost certainly present in the web data that large models train on.

Detecting contamination is difficult because model developers rarely have full transparency into what specific content was included in training data. Some researchers argue that canonical benchmark pass rates are significantly biased upward because of contamination and that only carefully designed held-out benchmarks can produce trustworthy numbers. Dynamically generated benchmarks offer a partial solution: if problems are created after the model's training cutoff, they cannot appear in training data. The LiveCodeBench project takes this approach, continuously adding new problems from competitive programming platforms timed so that each problem was published after the training cutoff of the models being evaluated.

A practical consequence for evaluation practitioners is that newly released models often show performance on recently-introduced benchmarks that is harder to explain through capability alone, suggesting memorization of specific problems. When a new benchmark is introduced and a new model immediately achieves surprisingly high scores, contamination should be the first hypothesis to investigate, not the last.

The binary pass/fail criterion for each problem is strict in a way that does not always match practical use. In production, a function that correctly handles 99% of inputs but crashes on a rare edge case might be perfectly acceptable if that edge case never occurs in the deployment context. A function for processing user uploads might not need to handle inputs larger than 100 MB if the application enforces that limit at the network layer. The strictness of execution-based evaluation means that models appear worse than they are for tasks with tolerant correctness requirements.

Some researchers have explored partial credit metrics that count what fraction of test cases pass, but this approach has its own problem: it rewards code that is almost correct in a superficial way, like getting the first nine test cases right because they are simple while failing on the tenth edge case. The debate between binary correctness and partial credit metrics reflects a deeper question about what we are trying to measure. If we want to measure the probability that a generated function works correctly in a production system, binary correctness against a test suite with broad input coverage is the right metric. If we want to measure how much of the specification the model understood, partial credit might be more informative.

In practice, most evaluation uses binary correctness because it is simpler, harder to game, and closer to the behavior users care about. When deployed code fails, it fails: users do not receive partial credit for functions that almost work.