Code Evaluation › Code Benchmarks

Benchmarks operationalize evaluation: they define the problems, test cases, and scoring rules that allow comparison across models and across time. The code generation community has developed several influential benchmarks, each targeting different aspects of programming capability and reflecting different assumptions about what matters.

HumanEval, introduced by Chen et al. in 2021, is the benchmark that established pass@k as the standard metric for code evaluation. It consists of 164 hand-written Python programming problems, each with a function signature, docstring specification, and test suite. The problems are relatively simple, drawn from topics like string manipulation, list processing, basic arithmetic, and elementary algorithms.

HumanEval was designed to measure whether models could generate code that works, not just code that looks plausible. The choice to use execution-based testing was deliberate: the authors found that BLEU scores and other text similarity metrics correlated poorly with functional correctness, motivating the need for a better evaluation criterion. The benchmark was released alongside the Codex model, which was the first code-focused language model large enough and capable enough to make the benchmark usefully discriminating.

One important design decision was to hand-write the problems rather than scraping them from existing coding sites like LeetCode or HackerRank. This reduced the risk of benchmark contamination, where models perform well because they have seen the exact problems during training. Since HumanEval problems were written fresh by the researchers, they did not exist in the web corpus that the models trained on. This gave early HumanEval scores more credibility as measurements of learned capability.

The canonical measure for HumanEval is pass@1, typically at temperature 0 or low temperature. Early GPT-3-based models scored around 10-15% on pass@1. By 2022, Codex achieved 72% pass@1 with 100 samples per problem. By 2024, the best frontier models exceeded 90%, prompting the community to develop harder successors because a saturated benchmark provides no signal for distinguishing among top models.

HumanEval has become something of a lingua franca for the field: nearly every new code model reports its HumanEval pass@1, making it easy to compare models across papers and track progress over time. This standardization is valuable for the research community, but it also means that model development has increasingly optimized for this specific benchmark. When a benchmark becomes a target, it often ceases to be a reliable measurement of the underlying capability it was meant to proxy. This Goodhart's Law dynamic is a recurring problem in machine learning benchmarking, and HumanEval is not immune to it.

Mostly Basic Python Problems (MBPP) is a larger benchmark containing about 500 Python problems with test cases, drawn from programming exercises and beginner coding challenges. MBPP complements HumanEval by covering a wider range of problem types and testing models on more varied specifications, some written by crowdworkers rather than domain experts.

MBPP distinguishes between the full problem set and a curated "sanitized" subset that has been manually reviewed to verify unambiguous specifications and correct test cases. Researchers typically report results on the sanitized subset because the full set contains some problems with ambiguous specifications or incorrectly labeled test cases that would unfairly penalize correct solutions.

The crowdworker-written specifications in MBPP are intentionally less polished than those in HumanEval, which tests whether models can handle realistic, slightly informal descriptions of programming tasks. Many real programming requests from developers are underspecified or written in casual natural language rather than precise technical prose. MBPP's naturalistic specifications are arguably a better proxy for the kinds of requests a code assistant encounters in daily use, even if the underlying tasks are simple.

When researchers report results on both HumanEval and MBPP, the scores tend to correlate strongly: models that do well on one tend to do well on the other. This correlation is expected since both benchmarks focus on Python function-writing tasks. The value of running both is that MBPP's larger problem set and broader coverage reduce variance in the estimated pass rate.

The APPS benchmark (Automated Programming Progress Standard) is substantially harder than HumanEval or MBPP. It contains 10,000 programming problems drawn from competitive programming platforms like Codeforces and LeetCode, spanning three difficulty tiers: introductory, interview-level, and competition-level problems.

APPS tests algorithmic problem-solving rather than just function implementation. Problems require understanding complex specifications written in the style of competitive programming challenges, choosing appropriate data structures, and implementing non-trivial algorithms. At the competition difficulty level, problems require techniques like dynamic programming over bit masks, advanced graph algorithms including network flow and strongly connected components, and mathematical number theory like modular exponentiation and prime factorization.

Even the best models achieve relatively low pass rates on APPS competition problems, suggesting that high-level algorithmic reasoning remains a significant challenge for current code LLMs. Competition-level problems require the model to understand the specification and derive the correct algorithm from first principles, often an insight that takes experienced competitive programmers significant time and effort. The introductory and interview-level difficulty tiers are more tractable and give a better signal for models that have already saturated HumanEval and MBPP.

One important distinction between APPS and simpler benchmarks is the role of problem-specific insight. HumanEval problems can often be solved by pattern-matching against common Python idioms. APPS competition problems frequently require mathematical insight that is not derivable by pattern-matching, which is why they remain hard even for models that have memorized large amounts of code. This makes APPS a better test of algorithmic reasoning capability.

A significant weakness of HumanEval is that its test suites are small, typically containing only 7-8 test cases per problem. A study by Liu et al. that created HumanEval+, an augmented version of HumanEval with many additional test cases generated through LLM-based mutation and manual curation, found that models appearing to score 65% or higher on the original benchmark often dropped to 50% or lower on the augmented version. The drop is not because the models suddenly became worse; it is because better tests revealed bugs that the original tests missed entirely.

EvalPlus is a general framework for augmenting existing benchmarks with automatically generated test cases. The approach uses LLM-based mutation to generate many new inputs per problem, validates the expected outputs by running the ground-truth reference solution, and filters out invalid or duplicate cases. This approach can expose bugs in model-generated code that the original test suites missed because they lacked edge cases or inputs with unusual structure.

The practical lesson from EvalPlus is that benchmark pass rates without augmentation are an upper bound on true correctness, and the gap can be substantial for benchmarks with thin test suites. When reading benchmark results in research papers, it is worth asking how many test cases each problem has and whether the test suite was independently validated. A model scoring 90% on HumanEval and 75% on HumanEval+ is not two different models with different capabilities; it is the same model measured with different test coverage. The HumanEval+ number is the more trustworthy one because it is harder to pass through lucky test-case gaps.

SWE-bench (Software Engineering Benchmark) takes a fundamentally different approach from function-writing benchmarks. Instead of isolated function-writing problems, it presents models with real GitHub issues from open-source Python repositories and asks them to generate patches that resolve the issue. Correctness is measured by running the repository's full test suite after applying the generated patch.

SWE-bench tests a much richer set of capabilities than any function-writing benchmark:

• Code comprehension at scale: Understanding large, multi-file codebases with complex interdependencies, rather than a single function signature and docstring
• Fault localization: Identifying which files and functions are relevant to the reported bug, often by reading the issue description and connecting it to specific code paths
• Patch generation: Producing changes that integrate cleanly with existing code, respecting existing variable names, class hierarchies, and architectural patterns
• Test satisfaction: Writing code that passes tests the model has not seen and that were written by the original repository maintainers

SWE-bench is substantially harder than function-generation benchmarks. Early results showed that even strong models resolved fewer than 5% of issues. Subsequent work combining retrieval-augmented generation, execution feedback, and multi-agent orchestration workflows pushed this number to around 30-50% on the SWE-bench Verified subset, but many real-world issues remain out of reach for fully automated systems.

SWE-bench represents the direction evaluation is moving: away from toy functions and toward the messy, context-heavy problems that developers face in their daily work. The benchmark's reliance on real open-source projects also makes contamination harder, since the problems are drawn from issues filed after specific repository commits that post-date most training data cutoffs. A model that has learned to engineer software should perform well on SWE-bench; a model that has merely memorized solutions from competitive programming sites should not.

The gap between HumanEval performance and SWE-bench performance for the same models is striking and informative. A model that solves 90% of HumanEval problems might resolve only 20-30% of SWE-bench issues. This gap reveals that the skills tested by these two benchmarks are partially distinct. Function-writing skill does not automatically confer repository-navigation skill, fault-localization skill, or the ability to write patches that fit cleanly into an existing architecture. SWE-bench makes this gap visible.

Most early benchmarks focused on Python, showing both the language's popularity in machine learning and data science and the relative abundance of Python code in training datasets. MultiPL-E addresses this by translating HumanEval and MBPP problems into 18 programming languages, including Java, JavaScript, TypeScript, Go, Rust, PHP, Ruby, Racket, Lua, and others. This translation allows direct comparison of model capabilities across languages while controlling for problem difficulty.

MultiPL-E revealed significant and consistent performance variation across languages. Models perform better on Python than on less common languages, showing the distribution of programming content in training data. A language with a smaller web presence and fewer open-source repositories will have contributed less data to the model's training, resulting in lower code quality for that language. Languages like Racket or Lua, which are used in specific niches but lack the broad community presence of Python or JavaScript, show substantially lower pass rates.

This multilingual lens is practically important because enterprise software is not written in Python alone. A model deployed for a Java backend team or a Go infrastructure team needs to perform well in those languages, not just in Python. MultiPL-E gives teams a way to evaluate whether a model's Python capabilities translate to the languages they use in production. The answer, which varies significantly across languages and model sizes, has real implications for deployment decisions.

Beyond pass rates, MultiPL-E reveals interesting patterns in where models make language-specific mistakes. Models sometimes generate code that would be correct in Python but is syntactically invalid in the target language, or they use Python-style idioms in languages that have different conventions. These errors reveal the limits of cross-lingual transfer and the importance of language-specific training data.

LiveCodeBench takes a different approach to the contamination problem. Rather than constructing a fixed benchmark and hoping it stays clean, LiveCodeBench continuously adds new problems from competitive programming platforms, timed to problems that were published after recent model training cutoffs. This ensures that the benchmark always contains problems the model has not seen during training.

The continuous update approach comes with tradeoffs. Comparing models evaluated at different times requires careful normalization because the problem difficulty distribution changes as new problems are added. But for the core goal of measuring generalization rather than memorization, a living benchmark beats a static one as models get larger and training sets grow to cover more of the internet.

LiveCodeBench also tracks performance over time for specific model versions, which reveals how quickly benchmark contamination affects results. When a benchmark stays static, models trained on more recent data tend to score higher partly because the benchmark has entered their training distribution. LiveCodeBench's continuous updates make this harder by keeping problems fresh.