Code Evaluation › Code Evaluation: Functional Correctness, pass@k, and Benchmarks

Writing code is only half the problem. The other half is knowing whether that code is correct. For human developers, correctness is usually verified by running tests, reading the output, or reasoning through the logic carefully. But when a language model generates code at scale, you need a systematic way to measure how often it succeeds, how much trust to place in its outputs, and where its failures cluster. Without that measurement infrastructure, model development becomes guesswork.

This chapter is about code evaluation: the frameworks, metrics, and benchmarks that let you measure the quality of generated code. You will learn how functional correctness differs from surface-level similarity metrics, why the pass@k metric matters and how to compute it, what the major code benchmarks test and what they miss, and how evaluation is extending beyond simple correctness toward style, security, and human preference. Each of these topics has practical consequences for anyone building or deploying code generation systems.

Code evaluation sits at the end of the code generation pipeline. In the previous chapters we covered how code LLMs are trained, how they understand existing codebases, how they complete and generate code, and how execution feedback enables iterative refinement. Now we close the loop: given a model that generates code, how do you know how good it is? This question turns out to be considerably more subtle than it looks. The answer touches on measurement theory, software testing philosophy, benchmark design, and the practical question of what "good code" even means in a production context.