Code Generation: Docstring-to-Code, Test-Driven, Strategies

Generating code from natural language descriptions is one of the most commercially impactful applications of language models. A developer describes what they want in plain English, and a model produces a working implementation. What once required parsing a specification document, designing an algorithm, and translating it into syntax now happens in seconds. But the gap between generating code that looks plausible and generating code that works is large, and understanding that gap is what this chapter is about.

Code generation combines language understanding with formal reasoning. Unlike generating a paragraph of prose, generating code requires producing output that is simultaneously syntactically valid, semantically meaningful, and functionally correct. A poem with an awkward phrase is still a poem. A function with a missing return statement is broken. This asymmetry between prose generation and code generation motivates nearly every design decision we cover here.

Building on the foundations from Code LLM Training, Code Understanding, and Code Completion covered earlier in Part XLVII, this chapter focuses on the full-generation setting: producing complete functions, modules, or programs from high-level specifications rather than filling in a single line or block. The prior chapters established how models learn the structure of code and how they operate in interactive completion environments. Here, we ask: given only a description of what code should do, how well can a model produce it?

To appreciate where modern code generation stands, it helps to understand where it came from. Early attempts at automating code writing were rooted in formal methods: specification languages like Z notation or Alloy allowed developers to describe system properties precisely, and theorem provers or constraint solvers could synthesize implementations that satisfied those specifications. This approach worked for narrow domains but required extensive manual formalization and could not scale to the breadth of software that developers write.

The next wave came from program synthesis research, which aimed to infer programs from input-output examples. Given a handful of example transformations (a list of strings and their desired cleaned-up forms, for instance), a synthesizer would search over a space of programs to find one consistent with all examples. Tools like FlashFill in Microsoft Excel, which learns text transformations from examples, are perhaps the most successful consumer-facing products from this era. But synthesis remained brittle: it worked in carefully restricted domains with well-designed domain-specific languages.

Neural approaches changed the problem formulation. Instead of searching over program spaces with logical solvers, neural code generation frames the problem as conditional language modeling: given a natural language prompt, generate the most likely code sequence under a model trained on enormous corpora of paired text and code. The quality no longer depends on formal specification completeness but on the model's ability to infer intent from imprecise natural language and generalize from patterns in training data.

This shift from correctness guarantees to probabilistic generation is both the source of modern code generation's power and the root of its failure modes. Formal synthesis guarantees correctness by construction. Neural generation produces code that is usually right but sometimes subtly wrong, often in ways that are hard to detect without careful testing.

Code generation is a broad term. In this chapter, we focus on two primary formulations that represent the majority of real-world use cases.

The first is docstring-to-code generation, where the input is a natural language description embedded in or adjacent to a function signature, and the output is the complete function body. This setting is the basis of the HumanEval and MBPP benchmarks and captures the experience of using tools like GitHub Copilot or Claude to implement a function from a description.

The second is test-to-code generation (sometimes called test-driven generation), where one or more unit tests serve as the specification. The model must generate code that passes all tests. This formulation has appealing properties: the specification is precise, the evaluation is automatic, and the generated code can be verified without human judgment.

Both settings share the core challenge: the model must infer the intended behavior from an informal or semi-formal specification, translate that intent into executable logic, and produce output that satisfies constraints the specification may not have made explicit. The challenge is not merely syntactic. Any competent programmer knows Python syntax. The hard part is bridging the semantic gap between a natural language description and the precise computational procedure the description is trying to convey.

Several settings fall outside these formulations. Repository-level code generation, where the model must create entire modules consistent with an existing codebase, is a separate and harder problem. Agentic code generation, where a model iteratively writes, executes, observes, and revises code over many steps, is yet another. This chapter addresses the foundational setting of function-level generation as the cleanest and most thoroughly studied case, with explicit notes about how insights extend to these broader settings.

Docstring-to-code generation is the dominant evaluation setting for code generation models. The input is a function signature paired with a natural language description, and the model must produce the complete implementation. This section covers how the problem is set up, why ambiguity in specifications is the central challenge, and how bidirectional context from fill-in-the-middle training changes what the model can infer.

In docstring-to-code generation, the input to the model takes a form like:

The model must complete this with a correct implementation. This sounds simple, but the difficulty scales quickly. The docstring above is unusually clear. Real docstrings are often ambiguous, incomplete, or inconsistent with edge case behavior. The model must make choices about how to handle empty lists, lists with one element, lists with duplicate values, and so on, all from a specification that may not mention these cases.

The reason docstring-to-code is a productive research formulation is that it isolates the generation problem cleanly. The function signature provides type hints and naming conventions. The docstring provides intent. The rest is implementation. This structure makes it possible to evaluate generated code automatically by running test suites.

When we break down what a model must do to succeed at this task, the components are:

• Intent parsing: Understanding what the function is supposed to compute, including which words in the docstring correspond to specific algorithmic operations
• Edge case inference: Reasoning about what happens at the boundaries of the input space even when the docstring is silent
• Algorithmic selection: Choosing an appropriate algorithm or data structure when multiple options are available
• Syntax production: Generating code that is syntactically valid in the target language
• Semantic consistency: Ensuring that all parts of the implementation are logically consistent with each other

Models fail on each of these components in different ways. Intent parsing fails when domain-specific terminology or ambiguous phrasing leads the model astray. Edge case inference fails when the training distribution does not include similar patterns. Algorithmic selection fails when the simplest approach the model would normally use is not appropriate for the given constraints. Syntax production failures are rare in modern large models but can occur for unusual language constructs.

Natural language descriptions are inherently underspecified compared to formal specifications. When a docstring says "sort the list," it does not specify whether sorting should be in-place or return a new list. When it says "find the top-k items," it does not specify behavior when there are ties at the k-th position. When it says "validate the input," it leaves open what constitutes valid input and what should happen when validation fails.

Large language models handle this ambiguity in ways that reflect their training distribution. If the vast majority of Python code uses sorted() to return a new sorted list rather than list.sort() for in-place sorting, the model will tend to generate the former even when either would technically be acceptable. This statistical bias toward common patterns is generally helpful but creates subtle mismatches when the caller's intent diverges from the convention.

One practical response is prompt augmentation: adding explicit edge case descriptions to the docstring before passing it to the model. If you write "returns a new sorted list (does not modify the input)," the model is much more likely to generate the intended behavior. Research on instruction tuning has shown that models respond well to this kind of precision, provided the model has been trained on similar instruction styles.

A deeper issue is that docstring quality in real codebases varies enormously. Academic benchmarks like HumanEval use carefully written, relatively complete docstrings. Production code is messier. Studies examining code generation on real-world repository functions find that performance drops substantially when the docstrings are typical of what engineers write under time pressure: brief, sometimes incorrect, and often missing entire behavioral dimensions. This is the key reason why RAG-augmented code generation, where relevant context from the codebase is inserted into the prompt, has become a standard production technique. If the docstring is ambiguous but the surrounding code provides disambiguating context, the model can use that context.

Consider the following two docstrings for the same function.

Version A (minimal):

Version B (complete):

Both describe the same function, but Version A leaves open whether to sort the output, whether to preserve order, whether to modify in place, and how to handle empty input. A model generating from Version A might produce a sorted(set(items)) one-liner that is fast and concise but destroys ordering, while the caller expected order preservation. This is not a model failure in any deep sense; it is a specification failure that creates a mismatch.

The practical lesson is that engineers who rely on code generation tools consistently report that the quality of their prompts determines the quality of the output. The model is a highly capable engineer who needs complete requirements, not a mind reader.

As we covered in Code LLM Training, fill-in-the-middle (FIM) training teaches the model to generate code that connects a given prefix to a given suffix. In the docstring-to-code setting, FIM enables a different kind of generation: the model can see both the function signature and any code that follows the function in the file, using context from both directions to constrain its output.

This bidirectional context is powerful. If the function being generated is later called with specific arguments, the model can infer usage patterns. If there is a test file that imports the function and checks its output, FIM-trained models can use those tests as additional specification signal. Prefix-only models see only the code before the cursor position and must extrapolate from partial context.

In practice, the difference between FIM and prefix-only generation is most pronounced when the function signature or docstring is ambiguous and the surrounding code provides disambiguating context. For clearly specified functions, both approaches tend to produce similar results. The advantage of FIM is especially visible in larger codebases where the calling code makes requirements explicit even when the docstring does not.

The mechanism by which FIM provides bidirectional signal is worth understanding. During FIM training, the model learns to predict the middle of a sequence given both its prefix and suffix. At inference time, when the model receives a prefix (the function signature and docstring) and a suffix (code following the function), it is in precisely the setting it trained on. The model has learned to generate code that is consistent with both boundaries simultaneously, which is fundamentally more constrained than extrapolating from the prefix alone.

Test-to-code generation replaces natural language descriptions with executable unit tests as the specification. This section explains why tests make better specifications than docstrings, what the model must do to generalize beyond the provided tests, and how this formulation fits into test-driven development workflows.

Test-to-code generation treats unit tests as the complete specification. The model receives one or more test cases and must generate a function implementation that passes all of them. This formulation has a key advantage over docstring-to-code: the tests are executable and unambiguous. There is no question about whether the generated code is "correct" relative to a vague description. Either the tests pass or they do not.

A typical test-to-code input looks like this:

Given these tests, the model must generate a find_median function. The tests encode the edge case behavior explicitly. Empty lists might still be unspecified, but many common ambiguities are resolved. The even-length case is handled (return the average of the two middle values, yielding 2.5 for [1,2,3,4]). The single-element case is specified (return that element). The negative-number case is covered.

What makes tests particularly powerful as specifications is that they are machine-checkable. There is no room for the kind of interpretive flexibility that natural language allows. If the model generates a function that returns 2 for [1,2,3,4] instead of 2.5, the test fails immediately and the error is reported with full precision. This tight feedback loop is what makes test-to-code generation so well suited to automated pipelines.

The challenge in test-to-code is that tests are a subset of the function's full behavior. A model that generates code passing the provided tests may still fail on unseen inputs. This is not a flaw in the approach; it reflects a fundamental property of testing. No finite set of tests can fully specify a function over all possible inputs.

The model must generalize from the provided tests to the underlying intent. A model that memorizes exactly what the tests check will produce brittle code. A model that infers the general pattern (median computation with proper handling of even-length lists) will produce robust code.

This generalization requirement creates an interesting asymmetry. A sufficiently adversarial model could generate a function that special-cases the exact test inputs and returns the correct answer for each case while doing something completely wrong for all other inputs. In practice, this does not happen because language models are not adversarial reasoners: they learn to generate code that reflects the general pattern implied by the tests, not code that memorizes specific input-output pairs. But the fact that this is technically possible illustrates why test-to-code evaluation must always include hidden tests that the model never sees.

The quality of test-to-code generation also depends heavily on how informative the provided tests are. A test suite with good coverage of edge cases (empty inputs, boundary values, type edge cases) gives the model much more information than a test suite that only covers the happy path. Models presented with tests that cover the specified behavior tend to generate more robust implementations than models given minimal tests, even when the minimal tests are technically sufficient to define the function's behavior.

One underappreciated aspect of test-to-code generation is that tests can encode behavior and intent. Consider two different implementations of find_median that both pass the four tests above. One uses sorted() and index arithmetic. Another uses heapq.nsmallest(). Both are correct for the given tests. But a well-written model will prefer the sorted() approach because the tests themselves use small sorted lists and integer comparisons, which pattern-matches strongly to the sorting-based implementation in training data.

This implicit alignment between test style and implementation style is an underexplored dimension of test-to-code quality. Tests written in a way that reflects the intended algorithmic approach tend to produce better-aligned implementations than tests that are implementation-agnostic. This matters in practice when developers write tests with a specific implementation strategy in mind and want the generated code to reflect that strategy.

Test-to-code generation aligns naturally with test-driven development (TDD), where tests are written before implementation. In a TDD workflow, the developer writes tests that specify desired behavior, and then a code generation model (or the developer themselves) implements the function. The model's output is immediately verifiable.

This integration changes how developers interact with code generation tools. Instead of prompting with a description and hoping the output is correct, the developer writes tests first, then generates an implementation, then runs the tests to verify. The feedback loop is tighter and the verification is automatic.

TDD-integrated code generation also changes the economics of code review. When generated code is paired with tests that the reviewer can run, reviewing becomes much faster than evaluating unconstrained generated code. The reviewer can verify behavior empirically rather than reasoning about correctness from first principles. Several engineering teams have reported that this combination, writing tests manually then generating implementations automatically, dramatically speeds up development for well-understood functionality while preserving human judgment for the test design itself, which requires deeper domain understanding.

The limitation of TDD-integrated generation is that writing good tests is itself a non-trivial skill. Tests that adequately specify complex behavior require the developer to think carefully about edge cases, invariants, and failure modes. In some sense, the work is not eliminated but redistributed: from writing implementations to writing tests. Whether this is preferable depends on the domain and the complexity of the code.

Given a trained model, the generation strategy determines how the model's probability distribution over tokens gets converted into actual code. Four strategies are particularly important: temperature control, nucleus sampling, best-of-N selection, and iterative self-repair. Each offers a different trade-off between quality, diversity, and compute cost.

Code generation quality depends heavily on how the model samples from its output distribution. At temperature 0 (greedy decoding), the model always chooses the highest-probability token. This produces deterministic, conservative outputs that reflect the most common patterns in the training distribution. Greedy decoding works well for straightforward functions where the standard implementation is clearly dominant.

At higher temperatures, the model introduces randomness, exploring less likely token sequences. This can be beneficial when the highest-probability path leads to an incorrect solution, as sampling may discover an alternative approach that works. However, high temperatures also increase the probability of syntactically invalid or logically incorrect code.

The temperature $T$ modifies the output probability distribution over the vocabulary. Given the model's raw logit scores $z_1, z_2, \ldots, z_V$ for $V$ vocabulary tokens, the probability assigned to token $i$ under temperature $T$ is:

where:

• $z_i$: the raw logit score produced by the model for token $i$
• $T$: the temperature parameter, controlling the sharpness of the distribution
• $\exp(z_i / T)$: the temperature-scaled exponential, amplified when $T < 1$ and flattened when $T > 1$

When $T \to 0$, the distribution concentrates all probability mass on the token with the highest logit (greedy decoding). When $T = 1$, the distribution is the model's natural softmax output. When $T > 1$, the distribution flattens, making low-probability tokens more likely.

The temperature trade-off is particularly important in code generation because the cost of errors is asymmetric. A slightly less common but correct implementation is far better than the most common incorrect one. This pushes practitioners toward using lower temperatures (typically 0.2 to 0.5) for code generation compared to higher temperatures used for creative text generation. The conventional wisdom is that code generation favors lower temperatures not because diversity is bad but because the quality gap between the most likely token and the second-most-likely token is often larger in code than in natural language, especially at structural decision points.

Nucleus sampling (top-p sampling) restricts the sampling pool to the smallest set of tokens whose cumulative probability exceeds a threshold $p$. At each step, the model constructs a probability distribution over the vocabulary, sorts by probability descending, and samples from the top-$p$ fraction of the mass.

The advantage of nucleus sampling over top-k sampling for code is that it adapts to the entropy of the distribution at each step. When the model is confident (low entropy, one token dominates), nucleus sampling selects from a small pool. When the model is uncertain (high entropy, many tokens have similar probability), it samples from a larger pool. This behavior is appropriate for code: at deterministic points (a closing bracket, an obvious variable name), confidence is high; at creative points (choosing an algorithmic approach), uncertainty is higher.

Top-k sampling, by contrast, always samples from exactly $k$ tokens regardless of the shape of the distribution. If $k = 40$ and the top token has probability 0.95, the model still samples from 40 options even though 39 of them are nearly impossible. Nucleus sampling avoids this problem by adapting the sampling pool to the distribution's natural shape.

In practice, nucleus sampling with $p$ between 0.90 and 0.95 is a common default for code generation, often combined with a temperature below 1.0. The combination of temperature scaling and nucleus sampling gives practitioners two independent levers: temperature controls the overall sharpness of the distribution, and the nucleus threshold controls how far into the tail the model is allowed to sample.

A straightforward and effective strategy for improving code generation quality is to generate multiple candidate solutions and select the best one. The selection can be done by:

• Running tests and returning the first solution that passes all of them
• Using a reward model trained to score code quality
• Asking a separate model to evaluate which candidate is most likely correct
• Combining automated tests with human review for critical applications

The mathematical justification for best-of-N is clear. If the probability that a single generation is correct is $p$, then the probability that at least one of $N$ independent generations is correct is:

where:

• $p$: probability that a single generation is correct
• $N$: number of candidate generations
• $(1-p)^N$: probability that all $N$ generations are incorrect

If $p = 0.3$ (a single generation is correct 30% of the time), then with $N = 5$ candidates, the probability that at least one is correct rises to $1 - (0.7)^5 \approx 0.83$. This is the intuition behind the pass@k metric, which we cover in the Code Evaluation chapter.

Best-of-N is computationally expensive, requiring $N$ times as many inference passes. But for many use cases, especially batch processing where latency is less critical than quality, the quality gains justify the cost. The strategy also has a pleasant property: it is trivially parallelizable. All $N$ generations can run simultaneously on $N$ inference servers, with no increase in wall-clock time if compute is available.

The key design question for best-of-N is the selection criterion. If executable tests are available, test execution is the most reliable selection strategy: generate $N$ candidates, run all provided tests on each, and select any candidate that passes. If no tests are available, selection must use a proxy quality signal, such as a reward model score or a secondary model's evaluation. Research comparing these selection strategies consistently finds that test-based selection outperforms model-based selection when the tests have good coverage, but model-based selection can be competitive when test coverage is sparse.

Code generation models can be used iteratively. After generating an initial solution, the model (or a separate model) can analyze any errors and generate a corrected version. This self-repair approach treats code generation as a multi-turn dialogue rather than a single forward pass.

The typical self-repair loop works as follows. The model generates an initial implementation. The code is executed (or statically analyzed). Any errors, including execution errors, failed tests, type errors, and linting warnings, are collected into an error message. The model receives the original specification, the generated code, and the error message, then generates a corrected implementation.

This approach is powerful because execution feedback is highly informative. If the code fails with AssertionError: expected 2.5, got 2.0, the model receives essentially a test oracle: it knows the specific input that failed, the expected output, and the actual output. This is far more specific than the original docstring, and it directly constrains the repair. Off-by-one errors, missing edge case handling, incorrect type coercions, and import errors all produce diagnostic messages that give the model precise guidance about what to fix.

Research has shown that self-repair can substantially improve pass rates on coding benchmarks, particularly for problems where the initial attempt is close but fails due to small errors like off-by-one mistakes, incorrect edge case handling, or missing imports. The key insight is that execution feedback provides a strong signal that is absent in the original generation pass.

An important subtlety is that self-repair works best when the initial generation is close to correct. If the initial generation has a fundamental algorithmic error (using an approach that cannot be fixed incrementally), repair loops tend to produce increasingly convoluted code that wraps the wrong approach in defensive logic rather than replacing it with a correct approach. Detecting when to discard the initial generation entirely versus when to repair it is an open research problem.

The number of repair rounds that provide meaningful benefit is typically small. Most studies find that the first repair round provides the largest gain, the second round provides additional but diminishing gains, and third and subsequent rounds provide little benefit. The explanation is that the errors that survive the first repair round are typically deeper algorithmic issues that require a fundamentally different approach, not surface-level fixes that iterative repair can address.

Structured decoding constrains the model's output to conform to a formal grammar or schema. For code generation, this can mean enforcing syntactic validity by restricting which tokens are legal at each position based on a partial parse of the output so far. A partially decoded Python function cannot end with a dangling colon; structured decoding would exclude tokens that would produce this invalid state.

The benefit of structured decoding is eliminating entire classes of syntax errors before they occur. A decoder that knows the grammar of the target language can guarantee that every generated sequence is syntactically valid, even if functional correctness remains probabilistic. This is particularly useful for lower-capability models or for generating code in languages with strict syntactic constraints.

The cost of structured decoding is engineering complexity: maintaining a live parser state alongside the decoding process requires integration with the language runtime and can slow inference. Several libraries now provide this capability for common languages, including Outlines for Python and TypeScript. These libraries integrate with popular model serving frameworks and handle the bookkeeping of tracking grammar state across generation steps.

Structured decoding represents a productive point of contact between the neural and classical approaches to code generation. Neural generation provides the semantic content; formal grammars provide syntactic guarantees. The combination addresses one of the clearest failure modes of pure neural generation while preserving the flexibility that makes neural approaches powerful.

Let us trace through the complete generation process for a concrete example, examining what can go wrong at each stage and how different strategies address those failures. This grounded walkthrough connects the abstract concepts above to the practical reality of code generation.

The problem is to generate a function find_median(numbers) that returns the median of a list of numbers. The median is the middle value when the list is sorted; for even-length lists, it is the average of the two middle values.

A minimal docstring like "Return the median of a list" might produce this output:

The complete docstring forced the model to address the even-length case explicitly.

Even the improved version may not handle the empty list case. Suppose the caller adds:

The repair added explicit empty-list handling. The error message was precise enough to tell the model exactly what failed, and the model correctly chose to raise ValueError rather than return a default value, following Python convention.

For a problem this simple, best-of-N is not strictly necessary. But consider what happens if the model generates multiple candidates at temperature 0.5:

• Candidate 1: Correct implementation using sorted()
• Candidate 2: Incorrect implementation using list.sort() in-place that breaks the original list
• Candidate 3: Correct implementation using statistics.median()
• Candidate 4: Correct implementation with explicit type conversion to float
• Candidate 5: Incorrect implementation that returns the wrong middle index

Running tests on all five candidates immediately identifies candidates 2 and 5 as incorrect and selects from the correct alternatives. The selection criterion is test execution, which takes milliseconds and requires no human judgment. This example illustrates the core value of best-of-N: even when individual generation quality is imperfect, generating multiple candidates and selecting with a reliable oracle produces high-quality output efficiently.

Quality in code generation is multi-dimensional, and the dimensions that matter depend on the context. Benchmarks focus on functional correctness because it can be measured automatically. Production deployments care about security, efficiency, and style as well. This section examines the full quality picture and two specific failure modes that are easy to miss.

Evaluating code generation quality requires thinking clearly about what "good" means. The dimensions are:

• Functional correctness: Does the code compute the right answer for all valid inputs?
• Robustness: Does it handle edge cases, invalid inputs, and boundary conditions gracefully?
• Readability: Is the code understandable to a human reviewer?
• Efficiency: Does it run in reasonable time and space?
• Style consistency: Does it match the conventions of the surrounding codebase?

These dimensions can trade off against each other. A highly optimized implementation may be harder to read. A highly readable implementation may be less efficient. The right balance depends on the use case.

In benchmarking research, functional correctness is almost exclusively the focus, measured by automated test suites. This is a practical choice: functional correctness can be measured automatically at scale. The other dimensions require human judgment or more sophisticated analysis tools. But dismissing the non-correctness dimensions entirely leads to a distorted picture of code generation utility. Code that is functionally correct but unreadable, inconsistent with the codebase style, or subtly inefficient still imposes maintenance costs.

One empirical finding in code generation research is that longer generated code is not necessarily more correct. Models sometimes generate verbose implementations with unnecessary logic, repeated computations, or defensive checks that introduce bugs while appearing thorough. A common failure mode is generating code that handles a specific case the model "worried about" but introduces an error in doing so.

Conversely, overly terse implementations sometimes omit necessary logic. The sweet spot is an implementation that matches the complexity of the problem: simple problems get simple implementations, complex problems get appropriately structured code.

This observation has implications for how we prompt code generation models. Asking for "a robust implementation" may produce verbose, error-prone code. Asking for "a concise implementation" may produce missing edge case handling. Being explicit about the trade-off ("implement this function concisely; edge cases are not needed for this use case") often produces better results.

The verbosity problem becomes more pronounced for longer functions. Research on models generating complete files or modules finds that output quality degrades as length increases, both because the model's effective attention to the specification weakens over distance and because longer generations accumulate more opportunities for individual token errors to compound into logical failures. This is one reason why function-level generation remains the most reliable setting.

Language models generate code character by character without maintaining a type environment or symbol table. This means they can produce code that is syntactically valid but type-incorrect, using variables of the wrong type, calling methods that do not exist on an object, or returning values of the wrong type.

Type annotations in the function signature help because they appear in the model's context and constrain the generation. If the signature declares def process(items: list[str]) -> dict[str, int]:, the model has type information to work with. Without annotations, the model must infer types from names and context alone.

Static analysis tools (mypy, pyright) can be used as part of a verification pipeline to detect type errors in generated code before execution. This is particularly valuable in codebases with strict type policies where type errors are unacceptable even if the code would run correctly without type checking.

The deeper issue is that type correctness is only the most formalized dimension of a broader consistency problem. Generated code can be semantically incorrect without any type error: it might return a value in the wrong units, use a dictionary key that was not established elsewhere in the codebase, or violate an invariant that is enforced by convention rather than by type annotations. Catching these semantic consistency failures requires either running tests that cover the specified behavior or reviewing the generated code carefully, and there is no fully automated solution.

A failure mode specific to code generation (and largely absent from prose generation) is hallucinated API calls: the model generates syntactically plausible function calls to methods or attributes that do not exist. This happens because the model has learned the statistical patterns of library usage from training data, but its knowledge of specific API versions may be outdated or incomplete.

For example, a model might generate a call to a DataFrame method using a parameter name that was renamed in a version update, or it might generate a call to a utility function from an internal library it has seen referenced in similar contexts without having access to the actual implementation. These errors are difficult to detect from static analysis alone because the method names and parameter patterns look plausible. They typically only surface at runtime.

The practical mitigation is to provide API documentation or type stubs in the model's context for the specific libraries being used. When the model can see the actual method signatures and docstrings, hallucination rates drop substantially. Several production code generation systems build tooling to automatically inject relevant API documentation into the prompt based on the imports in the surrounding code.

This section walks through practical implementations of docstring-to-code evaluation, sampling strategies, and a simple self-repair loop. We use standard Python for execution and verification, keeping the focus on the core logic rather than model API calls.

The core evaluation metric for code generation is pass@k: given $k$ candidate solutions per problem, what fraction of problems is solved by at least one candidate? We implement this metric directly.

The mathematically unbiased estimator for pass@k, as described in the HumanEval paper, avoids the variance of direct sampling by using the formula:

where:

• $n$: total number of generated samples per problem
• $c$: number of correct samples among the $n$
• $k$: the $k$ in pass@k (the number of attempts)
• $\binom{n-c}{k}$: ways to choose $k$ from the $n-c$ incorrect samples
• $\binom{n}{k}$: ways to choose any $k$ from all $n$ samples

This estimator is unbiased because it considers all possible ways of selecting $k$ samples from the $n$ generated, rather than directly sampling $k$ times, which would have higher variance. The formula computes the expected fraction of $k$-sample draws that contain at least one correct sample, which is exactly the quantity pass@k is trying to measure.

To see why this is useful: if you generate 10 candidates per problem and 4 are correct, you do not want to compute pass@5 by randomly drawing 5 candidates and checking if any are correct (this would have high variance). Instead, the closed-form expression tells you exactly what fraction of all possible 5-candidate draws would include at least one correct solution.

The pass@k scores show a clear trend: with more attempts per problem, performance improves substantially, showing why best-of-N selection is so effective in practice.

We can visualize how pass@k improves with more samples for problems at different difficulty levels. This helps build intuition for why generating multiple candidates is such a cost-effective way to improve output quality.

The curves make clear why best-of-N is so valuable. A problem that only 25% of individual generations solve is solved about 76% of the time with five independent candidates and crosses 90% at nine candidates. The diminishing returns are also visible: for easy problems (p=0.7), the second candidate raises the solve rate from 70% to 91%, while later samples add progressively less.

Temperature directly controls the diversity of generated code. We can demonstrate this by sampling tokens from a synthetic probability distribution and observing how temperature changes the distribution of selections.

At low temperature, the two leading tokens dominate the distribution. At high temperature, sampling spreads across many tokens, so lower-ranked continuations become more common. For code generation, that extra diversity can reveal alternative implementations, but it also raises the risk of syntactic or logical errors.

A self-repair loop generates code, tests it, and iteratively fixes any errors. We demonstrate the core logic using a synthetic problem where the "model" has a known error rate that decreases with each repair attempt.

The stylized example illustrates the pattern reported in self-repair systems: the first repair can fix many obvious failures, while later attempts operate on a progressively harder residual set. The exact gains depend on the model, tests, and feedback quality; the plotted rates are illustrative rather than benchmark measurements.

The quality of the natural language specification has a measurable effect on generation quality. We can simulate this by scoring docstrings for specificity and showing the relationship between docstring quality scores and generation success rates.

The positive trend supports the practical intuition that more specific docstrings help, while the scatter shows that specification quality is not the only factor. Clear edge cases, return behavior, and type constraints reduce ambiguity and can lower iteration costs downstream.

The key parameters for code generation pipelines are:

• temperature: Controls output diversity. Values of 0.2 to 0.5 work well for code, balancing correctness with some variation for best-of-N selection.
• top_p (nucleus sampling threshold): Typical values of 0.90 to 0.95 keep high-probability tokens while allowing occasional exploration.
• n (candidates per problem): More candidates dramatically improve pass@k. Even n=5 roughly doubles solve rates on moderately difficult problems compared to n=1.
• max_tokens: Set large enough to avoid truncating function bodies. For typical function implementations, 256 to 512 tokens is usually sufficient.
• repair iterations: One or two repair rounds provide most of the gain; three or more have diminishing returns.

This section bridges the gap between benchmark evaluation and deploying code generation in real engineering workflows. The considerations here are not captured by HumanEval scores but matter enormously in practice.

Large language models for code generation are context-sensitive: the quality of generation depends heavily on what context is provided alongside the specification. In the benchmark setting, the specification is self-contained. In production, function implementations often depend on types, utilities, and conventions defined elsewhere in the codebase.

Effective context window management involves deciding which parts of the codebase to include in the prompt for a given generation task. Too little context leaves the model guessing about types and conventions. Too much context fills the model's window with irrelevant information that can distract from the specification. Several strategies have emerged for managing this trade-off.

Retrieval-augmented context fetches relevant code snippets based on semantic similarity to the specification. If the docstring mentions UserSession, the retrieval system finds the UserSession class definition and includes it in the prompt. This is particularly effective for proprietary codebases where the model has no training exposure.

File-level context includes the full contents of the file where the function will be placed, giving import statements, class definitions, and related functions as context. This is simple to implement and often highly effective.

Dependency-driven context traces the dependencies of the function being generated (based on the docstring or specification) and includes the relevant definitions. This is more expensive to compute but provides precisely the information the model needs.

The structure of the prompt matters as much as the content. Well-designed prompts for code generation typically include:

• Language specification: Making the target language explicit (even if the file extension implies it) reduces generation errors in multilingual models.
• Style guidelines: If the codebase follows specific naming conventions or structural patterns, stating them explicitly in the prompt produces more style-consistent output.
• Example functions: Providing one or two example functions from the same codebase in the prompt gives the model a template to follow, which is particularly useful for functions that follow a common pattern.
• Failure mode guidance: If there are specific failure modes to avoid ("do not use recursive implementations," "do not import external libraries"), stating them explicitly prevents the model from making choices it would otherwise default to.

The one-shot and few-shot approaches borrowed from natural language generation work just as effectively for code. A prompt that includes one or two correctly implemented example functions from the codebase before the target docstring gives the model a style and structure template to follow, and this almost always produces more consistent output than a zero-shot approach.

Code generation tools are not uniformly beneficial across all types of coding tasks. Understanding where they add the most value helps engineers allocate their time effectively.

Code generation adds the most value for:

• Boilerplate and pattern-driven code: Implementations that follow well-known patterns (CRUD operations, data transformation pipelines, standard algorithm implementations) are generated reliably because the training distribution covers them extensively.
• Utility functions: Small, self-contained functions with clear specifications are where code generation most closely matches production needs.
• Test generation: Generating unit tests for existing functions is often faster and more reliable than generating implementations, because tests have a more regular structure.
• Language translation: Converting code from one language to another is a strong use case because the specification (the source code) is completely unambiguous.

Code generation is less reliable for:

• Novel algorithms: Functions that require creative algorithmic design rather than pattern matching are harder for models, which default to the most common approach even when a less common one is needed.
• Security-critical code: As discussed below, generated code has measurably higher rates of certain vulnerability patterns.
• System-level code: Code that interacts with hardware, operating system interfaces, or low-level memory management is less represented in training data and more sensitive to subtle errors.
• Domain-specific logic: Functions that encode business rules specific to a single organization are unlikely to have training analogues, and the model must work from the docstring alone without statistical support.

Code generation models are powerful but have clear failure modes that become apparent with real-world use. The most common is specification ambiguity: when the docstring or tests do not fully specify the function's behavior, the model makes choices based on training distribution priors that may not match the developer's intent. This is not a failure of the model; it is a fundamental limitation of underspecified input.

A more subtle limitation is distributional mismatch. Models are trained on publicly available code, which has a particular distribution of programming patterns, library choices, and idiomatic styles. Internal codebases often differ substantially from public code. Functions that interact with proprietary APIs, use internal utility libraries, or follow company-specific conventions will be poorly generated by models that have never seen those patterns. This is why context-augmented generation (providing examples from the target codebase in the model's context) often improves quality significantly on internal codebases.

Long-range dependency is another persistent challenge. Simple functions with local logic are generated well. Functions that must maintain consistency with complex state, follow multi-step protocols, or interact with global configuration are harder. The model's effective context window limits how much of the surrounding codebase it can reason about simultaneously.

There is also a subtler limitation around logical consistency in longer code segments. When generating a short function, the model can maintain coherence between its opening and closing lines easily. As the function grows longer or spans multiple interacting components, the probability of internal inconsistencies increases. The model may initialize a variable in a way that conflicts with how it uses it fifty lines later, introduce a loop that accumulates values incorrectly, or return a result that does not account for a conditional branch established earlier in the function. These errors are not random; they tend to occur at the boundaries of the model's "working memory" within a single generation pass.

Deploying code generation in production introduces security concerns that are absent from benchmarking settings. Models can generate code with security vulnerabilities: SQL injection vulnerabilities, path traversal issues, inadequate input validation, weak cryptographic choices. Benchmark evaluation measures functional correctness on provided tests, which rarely include security test cases.

Several studies have found that code generated by large language models has measurably higher rates of certain vulnerability patterns than human-written code reviewed by security teams. This does not mean generated code should be avoided in production; it means it should be treated with the same (or higher) scrutiny as code from an untrusted source. Automated static analysis, security-focused linting, and code review remain essential.

The vulnerability patterns most commonly found in generated code are not exotic. They include using string formatting to construct SQL queries (enabling injection), not validating file paths before opening them, using deprecated or weak cryptographic functions, and not handling authentication errors securely. These patterns appear in publicly available code and are therefore represented in training data, which means the model learns to reproduce them in contexts where they are inappropriate.

The mitigation strategy is to treat security properties as part of the specification, not as implicit requirements. Prompts that explicitly state security requirements ("use parameterized queries, not string formatting," "validate and sanitize all file paths") produce more secure code. Static analysis tools that specifically check for security anti-patterns should be part of any production pipeline that incorporates generated code.

HumanEval and MBPP, the dominant benchmarks for code generation, consist of relatively short, self-contained function implementations with clear specifications. This is very different from the code that developers need generated in practice: functions that interact with frameworks, handle complex data schemas, implement multi-step protocols, or require domain-specific knowledge.

Models that achieve high pass@1 rates on HumanEval may perform much worse on internal task-specific benchmarks. When adopting code generation tools, running evaluations on a sample of your own tasks provides a much more reliable estimate of actual utility than benchmark scores. The code evaluation chapter covers benchmark design and interpretation in more detail.

The benchmark-reality gap also manifests in the distribution of difficulty. HumanEval problems have a specific difficulty profile, dominated by medium-difficulty algorithmic problems that a competent programmer could implement in a few minutes. Real codebases include a different distribution: many trivially easy functions (getters, setters, simple data transformations) that models handle well, some moderately complex functions similar to HumanEval problems, and a non-trivial fraction of highly domain-specific functions that are much harder than anything in HumanEval. Benchmark scores disproportionately reflect performance on the medium tier. This means that benchmark scores underestimate performance on easy production tasks and overestimate it on hard ones.

The appropriate level of human oversight for generated code is a design choice that depends on the deployment context, the criticality of the code, and the quality of the testing infrastructure. Several patterns have emerged from engineering teams that deploy code generation at scale.

In low-stakes contexts with broad test coverage, generated code can be accepted automatically if all tests pass. This is common for generated tests themselves, for simple utility functions with clear specifications, and for boilerplate code in well-tested frameworks. In medium-stakes contexts, generated code is reviewed by the developer who requested it. The review is typically much faster than writing from scratch because the reviewer is verifying rather than creating.

In high-stakes contexts (security-critical paths, financial calculations, safety-relevant logic), generated code is reviewed with the same rigor as human-written code, possibly with additional scrutiny given the known limitations of generated code in these domains. Some teams treat generated code as always requiring a second reviewer regardless of apparent simplicity.

The key insight is that the appropriate oversight level is not a fixed property of code generation as a technology; it depends on the specific application context and the quality of automated verification available. Treating all generated code with maximum scrutiny is wasteful. Treating all generated code as equivalent to thoroughly reviewed human code is risky. Finding the right calibration for each use case is itself an engineering problem worth solving deliberately.

Code generation bridges the gap between natural language specification and executable implementation. Two primary formulations capture most practical use cases: docstring-to-code generation, where a natural language function description drives generation, and test-to-code generation, where unit tests serve as an executable specification.

Generation quality depends on both the model and the generation strategy. Key takeaways:

• Sampling temperature matters: use 0.2 to 0.5 for code generation rather than the higher values used for creative text
• Best-of-N selection dramatically improves pass@k rates; even generating five candidates instead of one roughly doubles solve rates on moderate problems
• Self-repair loops that incorporate execution feedback further improve correctness; one or two repair rounds provide most of the benefit
• Input specification quality is a major driver of output quality; clearer docstrings with edge case descriptions produce better code
• Benchmark scores are not the full picture: evaluate on representative samples of your own tasks
• Security and distributional mismatch are the most practically important limitations in production deployment
• Context management and prompt engineering are as important as model selection for production code generation quality

Code generation methods continue to change rapidly. Models are becoming more capable, context windows are growing, and tool-augmented generation (where the model can execute code, look up APIs, and inspect types during generation) is beginning to blur the line between single-pass generation and the agentic code generation systems of the near future. The foundations covered here, specification quality, sampling strategies, verification, and limitation awareness, remain relevant regardless of how the models themselves continue to improve.

The Code Execution chapter covers the execution environment and feedback mechanisms that make self-repair loops possible. The Code Evaluation chapter provides a deeper treatment of pass@k, HumanEval, and the methodological challenges of measuring code generation quality rigorously.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about code generation with language models.