Code Generation › Generation Strategies

Given a trained model, the generation strategy determines how the model's probability distribution over tokens gets converted into actual code. Four strategies are particularly important: temperature control, nucleus sampling, best-of-N selection, and iterative self-repair. Each offers a different trade-off between quality, diversity, and compute cost.

Code generation quality depends heavily on how the model samples from its output distribution. At temperature 0 (greedy decoding), the model always chooses the highest-probability token. This produces deterministic, conservative outputs that reflect the most common patterns in the training distribution. Greedy decoding works well for straightforward functions where the standard implementation is clearly dominant.

At higher temperatures, the model introduces randomness, exploring less likely token sequences. This can be beneficial when the highest-probability path leads to an incorrect solution, as sampling may discover an alternative approach that works. However, high temperatures also increase the probability of syntactically invalid or logically incorrect code.

The temperature $T$ modifies the output probability distribution over the vocabulary. Given the model's raw logit scores $z_1, z_2, \ldots, z_V$ for $V$ vocabulary tokens, the probability assigned to token $i$ under temperature $T$ is:

$$P_T(i) = \frac{\exp(z_i / T)}{\sum_{j=1}^{V} \exp(z_j / T)}$$

where:

• $z_i$: the raw logit score produced by the model for token $i$
• $T$: the temperature parameter, controlling the sharpness of the distribution
• $\exp(z_i / T)$: the temperature-scaled exponential, amplified when $T < 1$ and flattened when $T > 1$

When $T \to 0$, the distribution concentrates all probability mass on the token with the highest logit (greedy decoding). When $T = 1$, the distribution is the model's natural softmax output. When $T > 1$, the distribution flattens, making low-probability tokens more likely.

The temperature trade-off is particularly important in code generation because the cost of errors is asymmetric. A slightly less common but correct implementation is far better than the most common incorrect one. This pushes practitioners toward using lower temperatures (typically 0.2 to 0.5) for code generation compared to higher temperatures used for creative text generation. The conventional wisdom is that code generation favors lower temperatures not because diversity is bad but because the quality gap between the most likely token and the second-most-likely token is often larger in code than in natural language, especially at structural decision points.

Nucleus sampling (top-p sampling) restricts the sampling pool to the smallest set of tokens whose cumulative probability exceeds a threshold $p$. At each step, the model constructs a probability distribution over the vocabulary, sorts by probability descending, and samples from the top-$p$ fraction of the mass.

The advantage of nucleus sampling over top-k sampling for code is that it adapts to the entropy of the distribution at each step. When the model is confident (low entropy, one token dominates), nucleus sampling selects from a small pool. When the model is uncertain (high entropy, many tokens have similar probability), it samples from a larger pool. This behavior is appropriate for code: at deterministic points (a closing bracket, an obvious variable name), confidence is high; at creative points (choosing an algorithmic approach), uncertainty is higher.

Top-k sampling, by contrast, always samples from exactly $k$ tokens regardless of the shape of the distribution. If $k = 40$ and the top token has probability 0.95, the model still samples from 40 options even though 39 of them are nearly impossible. Nucleus sampling avoids this problem by adapting the sampling pool to the distribution's natural shape.

In practice, nucleus sampling with $p$ between 0.90 and 0.95 is a common default for code generation, often combined with a temperature below 1.0. The combination of temperature scaling and nucleus sampling gives practitioners two independent levers: temperature controls the overall sharpness of the distribution, and the nucleus threshold controls how far into the tail the model is allowed to sample.

A straightforward and effective strategy for improving code generation quality is to generate multiple candidate solutions and select the best one. The selection can be done by:

• Running tests and returning the first solution that passes all of them
• Using a reward model trained to score code quality
• Asking a separate model to evaluate which candidate is most likely correct
• Combining automated tests with human review for critical applications

The mathematical justification for best-of-N is clear. If the probability that a single generation is correct is $p$, then the probability that at least one of $N$ independent generations is correct is:

$$P(\text{at least one correct}) = 1 - (1 - p)^N$$

where:

• $p$: probability that a single generation is correct
• $N$: number of candidate generations
• $(1-p)^N$: probability that all $N$ generations are incorrect

If $p = 0.3$ (a single generation is correct 30% of the time), then with $N = 5$ candidates, the probability that at least one is correct rises to $1 - (0.7)^5 \approx 0.83$. This is the intuition behind the pass@k metric, which we cover in the Code Evaluation chapter.

Best-of-N is computationally expensive, requiring $N$ times as many inference passes. But for many use cases, especially batch processing where latency is less critical than quality, the quality gains justify the cost. The strategy also has a pleasant property: it is trivially parallelizable. All $N$ generations can run simultaneously on $N$ inference servers, with no increase in wall-clock time if compute is available.

The key design question for best-of-N is the selection criterion. If executable tests are available, test execution is the most reliable selection strategy: generate $N$ candidates, run all provided tests on each, and select any candidate that passes. If no tests are available, selection must use a proxy quality signal, such as a reward model score or a secondary model's evaluation. Research comparing these selection strategies consistently finds that test-based selection outperforms model-based selection when the tests have good coverage, but model-based selection can be competitive when test coverage is sparse.

Code generation models can be used iteratively. After generating an initial solution, the model (or a separate model) can analyze any errors and generate a corrected version. This self-repair approach treats code generation as a multi-turn dialogue rather than a single forward pass.

The typical self-repair loop works as follows. The model generates an initial implementation. The code is executed (or statically analyzed). Any errors, including execution errors, failed tests, type errors, and linting warnings, are collected into an error message. The model receives the original specification, the generated code, and the error message, then generates a corrected implementation.

This approach is powerful because execution feedback is highly informative. If the code fails with AssertionError: expected 2.5, got 2.0, the model receives essentially a test oracle: it knows the specific input that failed, the expected output, and the actual output. This is far more specific than the original docstring, and it directly constrains the repair. Off-by-one errors, missing edge case handling, incorrect type coercions, and import errors all produce diagnostic messages that give the model precise guidance about what to fix.

Research has shown that self-repair can substantially improve pass rates on coding benchmarks, particularly for problems where the initial attempt is close but fails due to small errors like off-by-one mistakes, incorrect edge case handling, or missing imports. The key insight is that execution feedback provides a strong signal that is absent in the original generation pass.

An important subtlety is that self-repair works best when the initial generation is close to correct. If the initial generation has a fundamental algorithmic error (using an approach that cannot be fixed incrementally), repair loops tend to produce increasingly convoluted code that wraps the wrong approach in defensive logic rather than replacing it with a correct approach. Detecting when to discard the initial generation entirely versus when to repair it is an open research problem.

The number of repair rounds that provide meaningful benefit is typically small. Most studies find that the first repair round provides the largest gain, the second round provides additional but diminishing gains, and third and subsequent rounds provide little benefit. The explanation is that the errors that survive the first repair round are typically deeper algorithmic issues that require a fundamentally different approach, not surface-level fixes that iterative repair can address.

Structured decoding constrains the model's output to conform to a formal grammar or schema. For code generation, this can mean enforcing syntactic validity by restricting which tokens are legal at each position based on a partial parse of the output so far. A partially decoded Python function cannot end with a dangling colon; structured decoding would exclude tokens that would produce this invalid state.

The benefit of structured decoding is eliminating entire classes of syntax errors before they occur. A decoder that knows the grammar of the target language can guarantee that every generated sequence is syntactically valid, even if functional correctness remains probabilistic. This is particularly useful for lower-capability models or for generating code in languages with strict syntactic constraints.

The cost of structured decoding is engineering complexity: maintaining a live parser state alongside the decoding process requires integration with the language runtime and can slow inference. Several libraries now provide this capability for common languages, including Outlines for Python and TypeScript. These libraries integrate with popular model serving frameworks and handle the bookkeeping of tracking grammar state across generation steps.

Structured decoding represents a productive point of contact between the neural and classical approaches to code generation. Neural generation provides the semantic content; formal grammars provide syntactic guarantees. The combination addresses one of the clearest failure modes of pure neural generation while preserving the flexibility that makes neural approaches powerful.