Code Evaluation: Functional Correctness and pass@k

Writing code is only half the problem. The other half is knowing whether that code is correct. For human developers, correctness is usually verified by running tests, reading the output, or reasoning through the logic carefully. But when a language model generates code at scale, you need a systematic way to measure how often it succeeds, how much trust to place in its outputs, and where its failures cluster. Without that measurement infrastructure, model development becomes guesswork.

This chapter is about code evaluation: the frameworks, metrics, and benchmarks that let you measure the quality of generated code. You will learn how functional correctness differs from surface-level similarity metrics, why the pass@k metric matters and how to compute it, what the major code benchmarks test and what they miss, and how evaluation is extending beyond simple correctness toward style, security, and human preference. Each of these topics has practical consequences for anyone building or deploying code generation systems.

Code evaluation sits at the end of the code generation pipeline. In the previous chapters we covered how code LLMs are trained, how they understand existing codebases, how they complete and generate code, and how execution feedback enables iterative refinement. Now we close the loop: given a model that generates code, how do you know how good it is? This question turns out to be considerably more subtle than it looks. The answer touches on measurement theory, software testing philosophy, benchmark design, and the practical question of what "good code" even means in a production context.

Evaluating code is harder than evaluating natural language text, and the difficulty comes from a fundamental asymmetry: there are many correct programs for any given specification, but there are also infinitely many incorrect ones. You cannot resolve this asymmetry by counting words or comparing strings.

For natural language generation, metrics like BLEU or ROUGE measure n-gram overlap between a generated string and a reference string. These metrics are imperfect proxies for quality, but they are at least sensible proxies when there is one canonical way to express an idea. Code is different. Two functions that use entirely different variable names, loop structures, and even algorithms can both be correct implementations of the same specification. A string-similarity metric will penalize both if they differ from the reference, even when both are functionally equivalent to the reference and to each other.

The alternative is execution-based evaluation: run the code, feed it test inputs, and check whether the outputs match. This sidesteps the equivalence problem completely. You do not need to compare structure or syntax. You just need to know whether the program behaves correctly on the cases you care about. This is the approach that dominates modern code evaluation, and it is the foundation on which everything else in this chapter builds.

Early work on code evaluation tried to get around the equivalence problem by asking human judges to rate code quality. This is accurate but slow and expensive. It also does not scale to the kind of continuous, automated evaluation that model development requires, where you might need to evaluate thousands of model outputs every time you adjust a training hyperparameter. Execution-based testing offers the best compromise: it is objective, automatable, and captures what matters about code from a user perspective. If the code runs correctly on the inputs you care about, that is what matters, regardless of how it looks.

Execution-based evaluation is more principled than string matching, but it has its own weakness: you can only test what you can test. A function that passes every test case in your suite might still fail on inputs you did not think to include. This is the test coverage problem, and it is fundamental to the structure of software testing, not an accidental limitation of any particular benchmark.

Consider a function that is supposed to sort a list. If your test suite only checks non-empty lists of integers, a buggy implementation that crashes on empty input or misbehaves on lists containing duplicates will appear correct. The model's pass rate on your suite is not the same as its true correctness rate, because the suite is an incomplete specification of what "correct sorting" means.

This gap matters because it affects how you interpret evaluation results. A model scoring 80% on a benchmark does not mean 80% of its outputs are correct programs in any absolute sense. It means 80% of the benchmark's test inputs are handled correctly by those programs. The gap between benchmark performance and production reliability depends entirely on how representative the benchmark's test cases are of the inputs the code will encounter in deployment.

The test coverage problem is well-known in traditional software testing, and it has generated an entire field of research around test generation, mutation testing, and formal verification. For code evaluation benchmarks, the practical implication is that benchmark designers must invest seriously in test suites with broad input coverage, and researchers must be appropriately skeptical of benchmark numbers when test suites are thin. We will see this tension recur when we examine specific benchmarks later in this chapter. The EvalPlus project made this concrete by showing that augmenting HumanEval's test suites substantially changes apparent pass rates, not because models got better or worse, but because better tests revealed bugs the original tests missed.

Even if a program passes all tests, it might still be unsuitable for production use. Evaluation researchers have identified at least four dimensions of code quality beyond functional correctness:

• Style and readability: Does the code follow the language's conventions, use meaningful names, and communicate intent clearly?
• Efficiency: Is the algorithm efficient, or does it use a quadratic loop where a linear one would serve equally well?
• Security: Does the code introduce SQL injection vulnerabilities, buffer overflows, or insecure cryptographic choices?
• Maintainability: Is the code well-structured, properly documented, and easy to modify when requirements change?

Most current benchmarks focus almost exclusively on functional correctness because it is the easiest dimension to automate. The richer dimensions require either human judgment or specialized static analysis tools. This is not a permanent limitation but a reflection of where the field currently is. We will discuss what is currently possible along each dimension at the end of this chapter, and the trend is clearly toward evaluation across more dimensions as the tools mature.

Before diving into execution-based evaluation in detail, it is worth understanding concretely why string-similarity approaches fail for code. This motivates the design choices that make execution-based benchmarks better and explains why the transition away from BLEU-style metrics was necessary.

Suppose a model generates this implementation of a function to find the maximum element in a list:

This version would score relatively high on BLEU because most tokens match the reference exactly, even though the function is incorrect for single-element lists (it raises an IndexError) and for lists where the maximum element is at index 0. Execution-based testing immediately catches this bug; BLEU rewards the match and cannot see the error.

This asymmetry, rewarding look-alike incorrect code and penalizing functionally equivalent correct code, is the fundamental problem with text similarity metrics for code. Execution-based evaluation resolves both issues simultaneously.

Functional correctness is the standard criterion for code evaluation: a program is correct if it produces the expected output for all valid inputs. The simplest way to measure this is unit tests. Given a function specification and a set of (input, expected output) pairs, you run the generated function on each input and check whether the output matches.

This binary pass/fail criterion is strict. A function that produces the right answer for 9 out of 10 test cases but crashes on the 10th is marked incorrect. This strictness is intentional: in real software, a function that fails on edge cases causes bugs that can be hard to diagnose and costly to fix. The fact that it works "most of the time" is often not the point. The function was called because it was needed, and if it fails on the specific input that arrives in production, the failure is total regardless of its historical success rate.

The strictness also has an important consequence for evaluation: a model's functional correctness score will always be a lower bound on its true capability. A model that generates a nearly correct solution, one with a subtle bug in an unusual edge case, gets zero credit on that problem even though it clearly understood the task. This is a real cost, but the alternative of awarding partial credit based on how many test cases pass creates its own problems, as we discuss in the Limitations section.

A single test suite paired with a natural language specification is called a problem in the benchmark literature. The model receives the specification, sometimes along with a few example input/output pairs for context, and must generate a function that passes a hidden test suite that it has not seen.

The key design choice in a problem is the test suite itself. Test cases need to cover the full range of inputs the function might reasonably encounter:

• Normal cases: Typical inputs that the function should handle without difficulty
• Edge cases: Boundary values like empty inputs, single elements, zero, or maximum representable values
• Error cases: Inputs that should trigger exceptions or error handling
• Performance cases: Large inputs that would expose quadratic time complexity or other scalability issues

Most public benchmarks provide a handful of test cases per problem, which is enough to distinguish correct from obviously wrong implementations but often not enough to catch subtle bugs in almost-correct implementations. The EvalPlus project, which we discuss in detail later, found that augmenting test suites from the typical 7-8 tests to hundreds of generated tests substantially changes the apparent pass rates of models. This demonstrates how much benchmark scores depend on test suite quality and how sensitive the numbers can be to measurement choices that are rarely reported prominently.

A well-designed benchmark should report how many test cases each problem has, how those test cases were generated, and whether they were manually reviewed for correctness. These details affect how much to trust the numbers but are often omitted from papers that simply report a final pass rate.

Running generated code requires a controlled execution environment. You cannot execute arbitrary generated code on your main system without risks: the code might access the filesystem, make network requests, spawn processes, consume unlimited resources, or enter infinite loops.

Practical evaluation systems use sandboxed execution with resource limits:

• Timeout limits: Kill any program that runs longer than a threshold, typically 1-10 seconds per test case. This catches infinite loops and prevents a single problem from consuming all evaluation time.
• Memory limits: Prevent programs from allocating excessive memory, which can crash the host system or interfere with other evaluations running in parallel.
• Process isolation: Use containers or virtual machines to prevent generated code from affecting the host system, other evaluation processes, or any sensitive data on the machine.
• Import restrictions: Block potentially dangerous imports like os, subprocess, or socket that could enable code to escape the sandbox environment.

The HumanEval benchmark, which we will examine shortly, uses a relatively simple approach: execute code in a subprocess with a timeout, catch any exceptions, and compare outputs. More recent and security-conscious benchmarks add tighter sandboxing using technologies like Docker containers, gVisor, or Firecracker microVMs.

Execution safety is a serious engineering challenge for large-scale evaluation. When you are running hundreds of thousands of model-generated programs to evaluate a model family, even a small fraction of adversarial or buggy programs that escape the sandbox can cause real damage. Production evaluation infrastructure at major AI labs typically uses full container isolation with fine-grained resource controls that go significantly beyond what academic benchmarks demonstrate. The academic benchmarks prioritize reproducibility and simplicity; production evaluation prioritizes safety.

The most important metric in code evaluation is pass@k: the probability that at least one of $k$ sampled solutions to a problem is functionally correct.

When you use a language model interactively for coding, you rarely commit to its first output without review. You might ask it to regenerate if the first attempt looks wrong, or you might use an integrated development environment that shows you multiple completions side by side and lets you choose. This realistic usage pattern motivates pass@k. If a model can solve a problem correctly on at least one of 10 tries, that is practically valuable even if it cannot solve it reliably on every first attempt.

pass@k also gives you a way to reason about model deployment. If you want to show the user one solution and be confident it is correct, you care about pass@1: the model needs to get it right the first time. If you are using the model to generate candidate patches that a human engineer will review before applying, pass@5 or pass@10 might be more relevant: you want the correct solution to appear somewhere in the candidates, so the engineer just has to recognize it rather than produce it from scratch. If you are running automated tests and accepting any solution that passes, pass@k for large $k$ captures the ceiling of what the model can achieve on that problem at all.

The metric was introduced explicitly to escape the limitations of evaluating only pass@1. Before pass@k, researchers would report the best-of-$k$ success rate: sample $k$ outputs, report 1 if any pass and 0 otherwise. This captures a similar intuition, but the estimator has high variance. If you happen to sample more or fewer correct solutions than expected by chance, your estimate shifts accordingly, making it hard to compare results reliably across papers.

The combinatorial pass@k formula provides a statistically principled way to measure the same quantity with substantially lower variance. Rather than committing to exactly $k$ samples, you generate a larger pool of $n$ samples and use the observed count of correct solutions to estimate the probability analytically. This design choice, more samples feeding an analytical estimator rather than fewer samples feeding a direct count, is what makes pass@k a reliable research metric.

Let $n$ be the total number of solutions sampled for a given problem, and let $c$ be the number of those solutions that pass the test suite. The probability that at least one of $k$ randomly chosen solutions, drawn without replacement from the $n$ samples, is correct equals one minus the probability that all $k$ chosen solutions are wrong.

The number of ways to choose $k$ solutions that are all wrong, drawing from the $n - c$ incorrect solutions, is $\binom{n-c}{k}$. The total number of ways to choose any $k$ solutions from all $n$ is $\binom{n}{k}$. So the probability that all $k$ chosen solutions are wrong is:

Taking the complement gives the pass@k estimator:

where:

• $n$: the total number of samples generated for the problem
• $c$: the number of correct samples (passing all tests) among the $n$ samples
• $k$: the number of samples shown to the user or tried in production
• $\binom{n-c}{k}$: the number of ways to choose $k$ samples that are all incorrect
• $\binom{n}{k}$: the total number of ways to choose any $k$ samples from $n$

This formula is only valid when $k \leq n$. When $n - c < k$, meaning there are fewer incorrect samples than the budget $k$, it is impossible to choose $k$ samples that are all wrong, so pass@k equals 1.0 exactly. The formula handles this as a boundary case.

An alternative definition would be: sample exactly $k$ solutions, run the tests, and report whether any passed. This is simpler but has high variance, especially when $k$ is small. If you happen to sample more correct solutions than expected by chance, you overestimate the model's pass@k. If fewer, you underestimate.

The combinatorial formula above provides an unbiased estimator of the true pass@k probability given that you have already sampled $n \geq k$ solutions and observed $c$ correct ones. By generating more samples than $k$, typically $n = 200$ for careful research evaluations, and evaluating pass@1, pass@10, and pass@100 all at once, you get stable estimates without rerunning the model multiple times.

This efficiency is significant. Running a large language model 200 times per problem is a substantial compute investment, but it is a fixed cost that gives you the entire pass@k curve. You can estimate pass@1, pass@10, and pass@100 all from the same set of 200 samples, with variance that decreases as $n$ increases. This makes systematic benchmark evaluation tractable even for researchers without enormous compute budgets, as long as they commit to the investment once per model per benchmark.

Suppose a model generates $n = 10$ solutions for a sorting problem, and $c = 3$ of them pass the test suite. Let us compute pass@1 and pass@5 explicitly.

For pass@1:

For pass@5:

So the model has only a 30% chance of solving the problem on a single attempt, but if you pick any 5 of its 10 solutions, there is a 91.7% chance that at least one is correct. This gap between pass@1 and pass@k for larger $k$ is typical: models often know the solution well enough to generate it sometimes, but not consistently enough to make every attempt reliable.

The ratio between pass@1 and pass@k at larger values reveals something important about the model's confidence and consistency. A model that achieves pass@1 = 0.90 and pass@10 = 0.99 is being consistent: its solutions are almost always correct, and sampling more of them adds only marginal value. A model that achieves pass@1 = 0.40 and pass@10 = 0.99 is diverse but inconsistent: it clearly knows how to solve problems because it can generate correct solutions, but it cannot do so reliably in a single attempt. These are very different behavioral profiles with different practical implications, and reporting pass@k at multiple values of $k$ lets you distinguish them in a way that a single number cannot.

The pass@k metric rewards diversity. If all $k$ samples are essentially the same solution with the same bug, they all fail together and pass@k is no better than pass@1. If the samples explore different algorithmic strategies, variable naming choices, and control flow patterns, at least one approach might succeed even if others fail. Diversity is what makes pass@k larger than pass@1.

This creates a meaningful tension in how to sample. Greedy decoding (temperature $T = 0$) produces the single most likely output and is appropriate for pass@1 measurements where you want the model's single best guess. Higher temperatures ($T = 0.6$ to $0.8$) produce more diverse outputs and are appropriate for pass@k measurements where $k > 1$, because diversity increases the chances that at least one sample is correct.

Most benchmark evaluations use temperature 0.8 for measuring pass@k with $k > 1$, and either temperature 0 or 0.2 for pass@1.

There is a deeper tradeoff here. High temperature improves pass@k for large $k$ but degrades pass@1, because increased randomness means the model's most reliable solution is less likely to be generated on any given attempt. Low temperature improves pass@1 but can harm pass@k, because samples cluster around the same output and diversity collapses. Finding the temperature that maximizes a specific pass@k value for a specific deployment scenario is itself an optimization problem. Production code assistant designers often tune temperature separately for interactive completion (where pass@1 matters most) and background generation pipelines (where pass@k for moderate $k$ is the goal).

Benchmarks operationalize evaluation: they define the problems, test cases, and scoring rules that allow comparison across models and across time. The code generation community has developed several influential benchmarks, each targeting different aspects of programming capability and reflecting different assumptions about what matters.

HumanEval, introduced by Chen et al. in 2021, is the benchmark that established pass@k as the standard metric for code evaluation. It consists of 164 hand-written Python programming problems, each with a function signature, docstring specification, and test suite. The problems are relatively simple, drawn from topics like string manipulation, list processing, basic arithmetic, and elementary algorithms.

HumanEval was designed to measure whether models could generate code that works, not just code that looks plausible. The choice to use execution-based testing was deliberate: the authors found that BLEU scores and other text similarity metrics correlated poorly with functional correctness, motivating the need for a better evaluation criterion. The benchmark was released alongside the Codex model, which was the first code-focused language model large enough and capable enough to make the benchmark usefully discriminating.

One important design decision was to hand-write the problems rather than scraping them from existing coding sites like LeetCode or HackerRank. This reduced the risk of benchmark contamination, where models perform well because they have seen the exact problems during training. Since HumanEval problems were written fresh by the researchers, they did not exist in the web corpus that the models trained on. This gave early HumanEval scores more credibility as measurements of learned capability.

The canonical measure for HumanEval is pass@1, typically at temperature 0 or low temperature. Early GPT-3-based models scored around 10-15% on pass@1. By 2022, Codex achieved 72% pass@1 with 100 samples per problem. By 2024, the best frontier models exceeded 90%, prompting the community to develop harder successors because a saturated benchmark provides no signal for distinguishing among top models.

HumanEval has become something of a lingua franca for the field: nearly every new code model reports its HumanEval pass@1, making it easy to compare models across papers and track progress over time. This standardization is valuable for the research community, but it also means that model development has increasingly optimized for this specific benchmark. When a benchmark becomes a target, it often ceases to be a reliable measurement of the underlying capability it was meant to proxy. This Goodhart's Law dynamic is a recurring problem in machine learning benchmarking, and HumanEval is not immune to it.

Mostly Basic Python Problems (MBPP) is a larger benchmark containing about 500 Python problems with test cases, drawn from programming exercises and beginner coding challenges. MBPP complements HumanEval by covering a wider range of problem types and testing models on more varied specifications, some written by crowdworkers rather than domain experts.

MBPP distinguishes between the full problem set and a curated "sanitized" subset that has been manually reviewed to verify unambiguous specifications and correct test cases. Researchers typically report results on the sanitized subset because the full set contains some problems with ambiguous specifications or incorrectly labeled test cases that would unfairly penalize correct solutions.

The crowdworker-written specifications in MBPP are intentionally less polished than those in HumanEval, which tests whether models can handle realistic, slightly informal descriptions of programming tasks. Many real programming requests from developers are underspecified or written in casual natural language rather than precise technical prose. MBPP's naturalistic specifications are arguably a better proxy for the kinds of requests a code assistant encounters in daily use, even if the underlying tasks are simple.

When researchers report results on both HumanEval and MBPP, the scores tend to correlate strongly: models that do well on one tend to do well on the other. This correlation is expected since both benchmarks focus on Python function-writing tasks. The value of running both is that MBPP's larger problem set and broader coverage reduce variance in the estimated pass rate.

The APPS benchmark (Automated Programming Progress Standard) is substantially harder than HumanEval or MBPP. It contains 10,000 programming problems drawn from competitive programming platforms like Codeforces and LeetCode, spanning three difficulty tiers: introductory, interview-level, and competition-level problems.

APPS tests algorithmic problem-solving rather than just function implementation. Problems require understanding complex specifications written in the style of competitive programming challenges, choosing appropriate data structures, and implementing non-trivial algorithms. At the competition difficulty level, problems require techniques like dynamic programming over bit masks, advanced graph algorithms including network flow and strongly connected components, and mathematical number theory like modular exponentiation and prime factorization.

Even the best models achieve relatively low pass rates on APPS competition problems, suggesting that high-level algorithmic reasoning remains a significant challenge for current code LLMs. Competition-level problems require the model to understand the specification and derive the correct algorithm from first principles, often an insight that takes experienced competitive programmers significant time and effort. The introductory and interview-level difficulty tiers are more tractable and give a better signal for models that have already saturated HumanEval and MBPP.

One important distinction between APPS and simpler benchmarks is the role of problem-specific insight. HumanEval problems can often be solved by pattern-matching against common Python idioms. APPS competition problems frequently require mathematical insight that is not derivable by pattern-matching, which is why they remain hard even for models that have memorized large amounts of code. This makes APPS a better test of algorithmic reasoning capability.

A significant weakness of HumanEval is that its test suites are small, typically containing only 7-8 test cases per problem. A study by Liu et al. that created HumanEval+, an augmented version of HumanEval with many additional test cases generated through LLM-based mutation and manual curation, found that models appearing to score 65% or higher on the original benchmark often dropped to 50% or lower on the augmented version. The drop is not because the models suddenly became worse; it is because better tests revealed bugs that the original tests missed entirely.

EvalPlus is a general framework for augmenting existing benchmarks with automatically generated test cases. The approach uses LLM-based mutation to generate many new inputs per problem, validates the expected outputs by running the ground-truth reference solution, and filters out invalid or duplicate cases. This approach can expose bugs in model-generated code that the original test suites missed because they lacked edge cases or inputs with unusual structure.

The practical lesson from EvalPlus is that benchmark pass rates without augmentation are an upper bound on true correctness, and the gap can be substantial for benchmarks with thin test suites. When reading benchmark results in research papers, it is worth asking how many test cases each problem has and whether the test suite was independently validated. A model scoring 90% on HumanEval and 75% on HumanEval+ is not two different models with different capabilities; it is the same model measured with different test coverage. The HumanEval+ number is the more trustworthy one because it is harder to pass through lucky test-case gaps.

SWE-bench (Software Engineering Benchmark) takes a fundamentally different approach from function-writing benchmarks. Instead of isolated function-writing problems, it presents models with real GitHub issues from open-source Python repositories and asks them to generate patches that resolve the issue. Correctness is measured by running the repository's full test suite after applying the generated patch.

SWE-bench tests a much richer set of capabilities than any function-writing benchmark:

• Code comprehension at scale: Understanding large, multi-file codebases with complex interdependencies, rather than a single function signature and docstring
• Fault localization: Identifying which files and functions are relevant to the reported bug, often by reading the issue description and connecting it to specific code paths
• Patch generation: Producing changes that integrate cleanly with existing code, respecting existing variable names, class hierarchies, and architectural patterns
• Test satisfaction: Writing code that passes tests the model has not seen and that were written by the original repository maintainers

SWE-bench is substantially harder than function-generation benchmarks. Early results showed that even strong models resolved fewer than 5% of issues. Subsequent work combining retrieval-augmented generation, execution feedback, and multi-agent orchestration workflows pushed this number to around 30-50% on the SWE-bench Verified subset, but many real-world issues remain out of reach for fully automated systems.

SWE-bench represents the direction evaluation is moving: away from toy functions and toward the messy, context-heavy problems that developers face in their daily work. The benchmark's reliance on real open-source projects also makes contamination harder, since the problems are drawn from issues filed after specific repository commits that post-date most training data cutoffs. A model that has learned to engineer software should perform well on SWE-bench; a model that has merely memorized solutions from competitive programming sites should not.

The gap between HumanEval performance and SWE-bench performance for the same models is striking and informative. A model that solves 90% of HumanEval problems might resolve only 20-30% of SWE-bench issues. This gap reveals that the skills tested by these two benchmarks are partially distinct. Function-writing skill does not automatically confer repository-navigation skill, fault-localization skill, or the ability to write patches that fit cleanly into an existing architecture. SWE-bench makes this gap visible.

Most early benchmarks focused on Python, showing both the language's popularity in machine learning and data science and the relative abundance of Python code in training datasets. MultiPL-E addresses this by translating HumanEval and MBPP problems into 18 programming languages, including Java, JavaScript, TypeScript, Go, Rust, PHP, Ruby, Racket, Lua, and others. This translation allows direct comparison of model capabilities across languages while controlling for problem difficulty.

MultiPL-E revealed significant and consistent performance variation across languages. Models perform better on Python than on less common languages, showing the distribution of programming content in training data. A language with a smaller web presence and fewer open-source repositories will have contributed less data to the model's training, resulting in lower code quality for that language. Languages like Racket or Lua, which are used in specific niches but lack the broad community presence of Python or JavaScript, show substantially lower pass rates.

This multilingual lens is practically important because enterprise software is not written in Python alone. A model deployed for a Java backend team or a Go infrastructure team needs to perform well in those languages, not just in Python. MultiPL-E gives teams a way to evaluate whether a model's Python capabilities translate to the languages they use in production. The answer, which varies significantly across languages and model sizes, has real implications for deployment decisions.

Beyond pass rates, MultiPL-E reveals interesting patterns in where models make language-specific mistakes. Models sometimes generate code that would be correct in Python but is syntactically invalid in the target language, or they use Python-style idioms in languages that have different conventions. These errors reveal the limits of cross-lingual transfer and the importance of language-specific training data.

LiveCodeBench takes a different approach to the contamination problem. Rather than constructing a fixed benchmark and hoping it stays clean, LiveCodeBench continuously adds new problems from competitive programming platforms, timed to problems that were published after recent model training cutoffs. This ensures that the benchmark always contains problems the model has not seen during training.

The continuous update approach comes with tradeoffs. Comparing models evaluated at different times requires careful normalization because the problem difficulty distribution changes as new problems are added. But for the core goal of measuring generalization rather than memorization, a living benchmark beats a static one as models get larger and training sets grow to cover more of the internet.

LiveCodeBench also tracks performance over time for specific model versions, which reveals how quickly benchmark contamination affects results. When a benchmark stays static, models trained on more recent data tend to score higher partly because the benchmark has entered their training distribution. LiveCodeBench's continuous updates make this harder by keeping problems fresh.

Let us implement the combinatorial pass@k estimator and apply it to a synthetic evaluation scenario. This will make the formula concrete and demonstrate the efficiency of computing multiple $k$ values from a single pool of samples.

With $n = 10$ total samples and $c = 3$ correct ones, pass@1 is 0.30 and pass@5 is 0.917. Adding more diversity by trying 10 samples guarantees at least one correct solution whenever there are any correct solutions at all, since if $c \geq 1$ and $k = n$, the probability that all $k$ chosen solutions are wrong requires choosing all of them from the $n - c$ incorrect ones, which is impossible when there are fewer incorrect samples than the budget.

In practice, you estimate pass@k for an entire benchmark by averaging the per-problem estimates:

The results show the expected pattern: pass@1 is the hardest floor (the model must be reliable on each attempt), and pass@k increases as $k$ grows because sampling diversity helps find correct solutions among the pool. The rate of increase depends on how many problems are in the "sometimes solved" category versus the "never solved" category.

A useful supplement to the overall pass@k score is a difficulty breakdown. Problems where $c = 0$ (no correct samples at all) represent complete failures: the model has no ability to solve these problems regardless of how many samples you take. Problems where $c = n$ represent trivially easy cases that the model solves reliably on every attempt. Problems in between, where $0 < c < n$, show the model's reliability gradient: it knows how to solve these problems sometimes but not consistently.

The plot below illustrates the temperature tradeoff: how increasing sampling temperature shifts the balance between reliability (pass@1) and capability (pass@10). Understanding this tradeoff is essential for configuring code generation systems for their specific deployment context.

Designing a code evaluation run well requires careful choices about several parameters that interact in non-obvious ways. Getting these right matters for obtaining reliable estimates and for making results comparable across papers.

The key parameters for code evaluation are:

• $n$ (samples per problem): More samples give a more stable estimate but cost proportionally more compute. $n = 200$ is common for careful research evaluations. For rapid iteration during model development, $n = 20$ or $n = 50$ is more tractable while still providing useful directional signal.
• $k$ (target budget): The deployment scenario determines which $k$ values matter. Pass@1 is the right metric for user-facing tools where the model must produce a correct answer immediately. Pass@10 or pass@100 is appropriate for pipeline systems where multiple candidates are generated and filtered by automated tests.
• Temperature: Lower temperatures for pass@1 measurements (exploit the model's best guess consistently), higher temperatures for pass@k with $k > 1$ (explore diverse solutions that increase the odds of at least one being correct).
• Test suite size: Larger test suites catch more bugs and give more trustworthy numbers. EvalPlus-style augmentation reduces the gap between benchmark performance and real-world correctness. When comparing models, prefer benchmarks that report test suite size per problem.
• Benchmark freshness: Older benchmarks are more likely to be contaminated by training data. LiveCodeBench and SWE-bench's reliance on post-cutoff content makes them more robust to contamination than static benchmarks built from problems that circulated online before model training.

Functional correctness is necessary but not sufficient for high-quality code. Evaluation researchers are extending the criteria in several directions, motivated by the observation that correctness alone does not predict whether code will be useful, maintainable, or safe in production.

A model might generate code that passes all tests but is poorly structured, hard to read, or contrary to the language's idioms. Several approaches to measuring code quality exist, and they capture different dimensions of the quality problem.

Static analysis tools like pylint and flake8 for Python, or ESLint for JavaScript, flag style violations, unused variables, overly complex functions, and other maintainability issues. Running these tools on model outputs gives a rough quality signal without human annotation. These tools were developed for human-written code, and they translate reasonably well to evaluating generated code, though generated code can sometimes trip their heuristics in unexpected ways. A model that generates highly nested code to avoid using a library function might trigger a "function too complex" warning even if the code is technically correct and readable.

Cyclomatic complexity measures how many independent execution paths exist through a function. A function with high cyclomatic complexity is hard to test exhaustively and difficult to maintain when requirements change. Models sometimes generate solutions with unnecessary nesting or redundant conditional branches where a cleaner boolean expression or an early return would suffice. This happens partly because training data includes a wide range of coding styles, some of which favor explicit branching over compact expressions.

Idiomatic code scoring compares model outputs against the distribution of real code in the language's standard libraries and popular open-source projects. A Python function that reimplements the behavior of itertools.groupby from scratch rather than importing it is technically correct but idiomatically poor, and any experienced Python developer would find it surprising. This approach requires a reference corpus and a way to compute distributional similarity, but it captures an important quality dimension that binary pass/fail testing misses entirely. Idiomatic code is easier to read, easier to maintain, and benefits from the testing and optimization that went into the standard library implementation.

Security bugs in generated code are a real and growing concern. Studies have shown that Copilot and similar tools can generate code containing CWE (Common Weakness Enumeration) vulnerabilities, including SQL injection through string interpolation, path traversal through unvalidated user input, and use of deprecated cryptographic functions with known weaknesses.

The SecurityEval benchmark tests models specifically on security-sensitive programming scenarios, asking them to generate code for tasks like reading files, making database queries, handling authentication, or processing user input. Each problem has ground truth labels for secure versus insecure patterns, allowing automated scoring of whether the generated code avoids known vulnerability classes.

Evaluating security thoroughly requires domain expertise and cannot be fully automated with current tools. Static analyzers can catch some vulnerability patterns, but sophisticated security issues often require understanding the deployment context rather than just the code itself. A function that reads a file whose path comes from user input might look fine in isolation but creates a directory traversal vulnerability when deployed in a web server context. Evaluating this correctly requires knowing the deployment context, which automated tools typically cannot infer from the function alone. This contextual gap means that security evaluation is necessarily incomplete for any purely automated benchmark.

Two functions can both be functionally correct but differ by orders of magnitude in runtime or memory usage. A bubble sort and a mergesort both sort lists correctly, but on 10,000 elements, the difference is hundreds of milliseconds versus a few milliseconds. On 1,000,000 elements, one function is still fast and the other never returns within any reasonable time limit.

Efficiency evaluation typically works by first verifying that a solution is correct, then measuring its runtime on large inputs and comparing against the fastest known solution for each problem. APPS competition problems naturally test algorithmic efficiency because the judge's time limits are set to require efficient implementations. A solution that passes the correctness check but times out on large inputs is counted as incorrect in the competition context, which directly incentivizes efficient algorithms.

For models deployed in performance-sensitive contexts, including embedded systems, high-frequency trading, and real-time data processing, efficiency is a critical evaluation criterion. A model that generates provably correct but quadratic solutions to problems that admit linear ones is providing less value, even if it scores well on correctness benchmarks. The efficiency dimension is underrepresented in current evaluation frameworks but represents a real gap between what benchmarks measure and what deployed systems require.

The visualization below illustrates why efficiency matters beyond correctness, showing how the runtime gap between algorithmic complexity classes grows dramatically with input size.

Beyond automated metrics, human evaluators can assess qualities that are hard to formalize: whether code is readable, whether the solution approach is clever or unnecessarily convoluted, whether comments are helpful or misleading, and whether the variable names communicate intent. Human evaluation is expensive and slow to scale, but it is often more predictive of real-world usefulness than automated metrics alone, because it captures overall quality in the way a working developer would experience it.

Some researchers have used pairwise preference studies: show two generated solutions to a developer, ask which they prefer, and aggregate preferences using Elo-style rankings. This captures overall quality without requiring evaluators to specify exactly what they are measuring. Human preference studies have revealed that developers weight code readability and clarity more heavily than automated metrics suggest, which points toward a gap between what benchmarks measure and what developers value when using code assistants in their daily work.

An important finding from human preference studies is that correctness and preference do not always align. Developers sometimes prefer elegant incorrect code over ugly correct code, at least until they run the tests. They also prefer solutions that match the idioms of the codebase they are working in, which is something that automated metrics cannot assess without access to the surrounding codebase context. This suggests that preference-based evaluation captures different information than pass-rate benchmarks and complements rather than duplicates them.

The most ambitious evaluations combine multiple criteria to capture the full complexity of code quality. CRUXEval (Code Reasoning Understanding and eXecution Evaluation) tests whether models can reason about code execution in both directions: given a function and an output, predict the input that produced it; or given a function and an input, predict the output. This tests understanding of code semantics rather than just generation ability.

CodeReviewer asks models to review existing code and identify bugs, then evaluates whether the identified bugs are real and the suggested fixes are correct. This tests a different capability than generation: the ability to critically analyze code written by someone else and provide useful feedback. Many professional developers spend as much time reviewing code as writing it, so this is a practically important capability that pure generation benchmarks miss.

The trend in evaluation is clear and consistent: the field is moving toward multi-dimensional evaluation suites that capture the full complexity of software engineering, rather than single-number benchmarks that reduce quality to a pass rate on function-writing exercises. This is the right direction conceptually, but it comes with coordination costs. As evaluation becomes more multidimensional, comparing models across papers becomes harder because different papers use different combinations of metrics, weight them differently, and make different choices about benchmark design. The field will need to develop conventions for multi-dimensional reporting similar to the conventions that developed around HumanEval pass@1.

Even the best current evaluation frameworks have significant gaps that the research community is actively working to address. Understanding these gaps helps you interpret benchmark numbers more accurately and design evaluation strategies appropriate for specific deployment contexts.

Benchmarks are built at a point in time, and both models and benchmarks evolve in ways that create distribution shift. A benchmark that was challenging when introduced becomes saturated quickly as models improve, and the community then needs to build harder successors. But harder benchmarks often move toward competitive programming challenges that test algorithmic puzzle-solving rather than practical software engineering, which widens the gap between benchmark performance and the tasks developers face.

The broader distribution shift issue is between benchmarks and real code in production. Developers do not spend most of their time implementing sorting algorithms or string manipulation functions. They write database queries, API integrations, infrastructure configuration, authentication code, test code, and documentation. They debug issues in systems with complex state and interleaved concerns. Most benchmarks significantly underrepresent these practical tasks, not because benchmark designers do not understand this gap but because these tasks are much harder to evaluate automatically.

A model that scores 90% on HumanEval and 5% on SWE-bench is exhibiting exactly this distribution shift problem: excellent at toy functions, much weaker at real-world engineering tasks. The gap is not a failure of evaluation methodology but a real difference in capability that different benchmarks reveal. Reporting only HumanEval numbers gives a misleading picture of model usefulness for actual software development work.

The visualization below shows the performance gap between function-level benchmarks and repository-level benchmarks for hypothetical model families, illustrating that benchmark score does not transfer linearly across task types.

Natural language specifications are inherently ambiguous, and this creates a category of evaluation errors that are not the model's fault. A docstring that says "sort the list" leaves open questions: ascending or descending? In place or return a new list? Stable sort or not? What should happen for inputs containing non-comparable types? Models that generate code satisfying one reasonable interpretation may fail tests written for another equally reasonable interpretation.

More careful benchmarks try to write unambiguous specifications, but ambiguity is difficult to fully eliminate from natural language. This is also realistic in an important sense: real programming specifications written by product managers, issue trackers, and coworkers are often underspecified, and a good code assistant should either ask for clarification or generate code that covers the most common interpretation and documents its assumptions. Some recent evaluation work has started studying ambiguity resolution explicitly, asking models to identify underspecified requirements and generate clarifying questions before writing code. This capability is arguably more valuable than the ability to implement clearly specified functions, yet it is almost entirely absent from current benchmarks.

If a model's training data contains the solutions to benchmark problems, or even the problem statements themselves, its performance overstates its true generalization ability. This is a serious concern for benchmarks built from problems that appear on public coding websites, because competitive programming problems and their solutions are almost certainly present in the web data that large models train on.

Detecting contamination is difficult because model developers rarely have full transparency into what specific content was included in training data. Some researchers argue that canonical benchmark pass rates are significantly biased upward because of contamination and that only carefully designed held-out benchmarks can produce trustworthy numbers. Dynamically generated benchmarks offer a partial solution: if problems are created after the model's training cutoff, they cannot appear in training data. The LiveCodeBench project takes this approach, continuously adding new problems from competitive programming platforms timed so that each problem was published after the training cutoff of the models being evaluated.

A practical consequence for evaluation practitioners is that newly released models often show performance on recently-introduced benchmarks that is harder to explain through capability alone, suggesting memorization of specific problems. When a new benchmark is introduced and a new model immediately achieves surprisingly high scores, contamination should be the first hypothesis to investigate, not the last.

The binary pass/fail criterion for each problem is strict in a way that does not always match practical use. In production, a function that correctly handles 99% of inputs but crashes on a rare edge case might be perfectly acceptable if that edge case never occurs in the deployment context. A function for processing user uploads might not need to handle inputs larger than 100 MB if the application enforces that limit at the network layer. The strictness of execution-based evaluation means that models appear worse than they are for tasks with tolerant correctness requirements.

Some researchers have explored partial credit metrics that count what fraction of test cases pass, but this approach has its own problem: it rewards code that is almost correct in a superficial way, like getting the first nine test cases right because they are simple while failing on the tenth edge case. The debate between binary correctness and partial credit metrics reflects a deeper question about what we are trying to measure. If we want to measure the probability that a generated function works correctly in a production system, binary correctness against a test suite with broad input coverage is the right metric. If we want to measure how much of the specification the model understood, partial credit might be more informative.

In practice, most evaluation uses binary correctness because it is simpler, harder to game, and closer to the behavior users care about. When deployed code fails, it fails: users do not receive partial credit for functions that almost work.

Code evaluation has evolved from approximate string-matching metrics to execution-based testing that directly measures functional correctness, and from single-metric benchmarks toward multi-dimensional evaluation suites that capture the full complexity of code quality.

The key ideas are:

• Functional correctness is the primary criterion: a function is correct if it passes all test cases, and execution-based evaluation is the right way to measure it. String similarity metrics like BLEU fail because semantically identical functions can look very different, and semantically different functions can look very similar.
• pass@k quantifies the probability that at least one of $k$ sampled solutions is correct. The combinatorial estimator avoids sampling variance by using $n \geq k$ samples and computing the result analytically using the complement rule.
• pass@1 vs. pass@k: pass@1 measures reliability (can the model solve this on the first try?), while pass@k measures capability (can the model solve this at all given multiple attempts?). Reporting both reveals whether a model is consistent or merely capable.
• Major benchmarks: HumanEval established pass@k as the standard and measures basic function-writing in Python. MBPP extends breadth with more varied, naturalistic specifications. APPS tests algorithmic problem-solving at competitive programming difficulty. SWE-bench tests real-world bug fixing in full repositories. MultiPL-E evaluates performance across 18 programming languages. LiveCodeBench avoids contamination through continuous problem updates.
• Benchmark limitations: Small test suites inflate pass rates. EvalPlus-style augmentation reveals the gap. Contamination can overstate generalization. Competitive programming problems underrepresent practical engineering tasks. The distribution shift between HumanEval and SWE-bench scores reveals that different benchmarks test different capabilities.
• Beyond correctness: Evaluation is expanding to cover code style and idiom, security vulnerabilities, execution efficiency, and human preference, because functional correctness is necessary but not sufficient for production-ready code. Each dimension requires different tools and reveals different gaps.

As we move into production systems, the evaluation metrics covered in this chapter become the basis for deployment decisions: how often to show model suggestions, when to require human review, and how to monitor model quality in live environments where ground truth is not immediately available. A model that solves 90% of HumanEval problems but only 25% of SWE-bench issues needs different deployment guardrails than one that performs well on both.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about code evaluation metrics and benchmarks.