Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Code Generation › Code Implementation

Michael BrenndoerferFebruary 3, 202611 min read

Language AI HandbookAll chapters

Part 47 of 67Code Generation

Chapter 360Code Generation › Code Implementation

Reading section 9 of 13 · Code Implementation · 11 minSection 9 of 13 · 11 min

Section 9 of 13 · 11 min read

Code Generation › Code ImplementationOne section · Sections in this chapter9 / 13 ↓

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

This section walks through practical implementations of docstring-to-code evaluation, sampling strategies, and a simple self-repair loop. We use standard Python for execution and verification, keeping the focus on the core logic rather than model API calls.

SetupLink Copied

Evaluating Code Generation Quality with Pass@kLink Copied

The core evaluation metric for code generation is pass@k: given $k$ candidate solutions per problem, what fraction of problems is solved by at least one candidate? We implement this metric directly.

The mathematically unbiased estimator for pass@k, as described in the HumanEval paper, avoids the variance of direct sampling by using the formula:

\text{pass@}k = 1 - \frac{\binom{n - c}{k}}{\binom{n}{k}}

where:

$n$ : total number of generated samples per problem
$c$ : number of correct samples among the $n$
$k$ : the $k$ in pass@k (the number of attempts)
$\binom{n-c}{k}$ : ways to choose $k$ from the $n-c$ incorrect samples
$\binom{n}{k}$ : ways to choose any $k$ from all $n$ samples

This estimator is unbiased because it considers all possible ways of selecting $k$ samples from the $n$ generated, rather than directly sampling $k$ times, which would have higher variance. The formula computes the expected fraction of $k$ -sample draws that contain at least one correct sample, which is exactly the quantity pass@k is trying to measure.

To see why this is useful: if you generate 10 candidates per problem and 4 are correct, you do not want to compute pass@5 by randomly drawing 5 candidates and checking if any are correct (this would have high variance). Instead, the closed-form expression tells you exactly what fraction of all possible 5-candidate draws would include at least one correct solution.

In[4]:

Code

import math

import numpy as np


def pass_at_k(n: int, c: int, k: int) -> float:
    """
    Compute the unbiased pass@k estimator.

    n: total samples generated
    c: number of correct samples
    k: k in pass@k
    """
    if n - c < k:
        return 1.0
    # Use log-space computation to avoid overflow with large n
    log_numerator = sum(math.log(n - c - i) for i in range(k))
    log_denominator = sum(math.log(n - i) for i in range(k))
    return 1.0 - math.exp(log_numerator - log_denominator)


def compute_pass_at_k_results(results, k_values):
    """
    Compute pass@k for a set of problem results.

    results: list of dicts with 'n' (total) and 'c' (correct) keys
    k_values: list of k values to compute
    """
    metrics = {}
    for k in k_values:
        scores = [pass_at_k(r["n"], r["c"], k) for r in results if r["n"] >= k]
        metrics[f"pass@{k}"] = np.mean(scores) if scores else 0.0
    return metrics

Out[5]:

Console

  pass@1: 0.382
  pass@2: 0.583
  pass@5: 0.841
  pass@10: 0.940

The pass@k scores show a clear trend: with more attempts per problem, performance improves substantially, showing why best-of-N selection is so effective in practice.

Simulating Best-of-N SelectionLink Copied

We can visualize how pass@k improves with more samples for problems at different difficulty levels. This helps build intuition for why generating multiple candidates is such a cost-effective way to improve output quality.

In[6]:

Code

def simulate_pass_at_k_curve(p_correct, max_k=20):
    """
    Compute theoretical pass@k for a problem where each
    attempt independently succeeds with probability p_correct.
    """
    return [1 - (1 - p_correct) ** k for k in range(1, max_k + 1)]

Out[7]:

Visualization

Four pass-at-k curves for single-sample solve rates of 0.10, 0.25, 0.50, and 0.70. Higher-rate curves cross a horizontal 90 percent threshold with fewer attempts. — Pass@k rises as more candidate solutions are generated. Problems with a 70 percent single-sample solve rate exceed 90 percent by the second attempt, while problems with a 10 percent rate remain below 90 percent after 20 attempts.

The curves make clear why best-of-N is so valuable. A problem that only 25% of individual generations solve is solved about 76% of the time with five independent candidates and crosses 90% at nine candidates. The diminishing returns are also visible: for easy problems (p=0.7), the second candidate raises the solve rate from 70% to 91%, while later samples add progressively less.

Temperature and Sampling StrategyLink Copied

Temperature directly controls the diversity of generated code. We can demonstrate this by sampling tokens from a synthetic probability distribution and observing how temperature changes the distribution of selections.

In[8]:

Code

def sample_with_temperature(logits, temperature, n_samples=1000):
    """
    Sample from a distribution parameterized by logits at a given temperature.
    Returns an array of sampled indices.
    """
    if temperature == 0:
        return np.full(n_samples, np.argmax(logits))

    scaled_logits = logits / temperature
    # Numerical stability: subtract max before exp
    scaled_logits = scaled_logits - np.max(scaled_logits)
    probs = np.exp(scaled_logits)
    probs = probs / probs.sum()
    return np.random.choice(len(logits), size=n_samples, p=probs)

Out[9]:

Visualization

Bar chart of token-selection probabilities at temperature 0.2, dominated by the first two ranked tokens. — At temperature 0.2, the two highest-logit tokens receive nearly all probability mass. Sampling is conservative but retains a visible alternative to the top-ranked token.

Bar chart of token-selection probabilities at temperature 1.5, with nonzero mass across all ten ranked tokens. — At temperature 1.5, probability mass spreads across the full candidate set. The flatter distribution increases diversity but also selects lower-ranked tokens more often.

At low temperature, the two leading tokens dominate the distribution. At high temperature, sampling spreads across many tokens, so lower-ranked continuations become more common. For code generation, that extra diversity can reveal alternative implementations, but it also raises the risk of syntactic or logical errors.

Implementing a Simple Self-Repair LoopLink Copied

A self-repair loop generates code, tests it, and iteratively fixes any errors. We demonstrate the core logic using a synthetic problem where the "model" has a known error rate that decreases with each repair attempt.

In[10]:

Code

def simulate_repair_round(attempt, base_pass_rate=0.45, seed_offset=0):
    """
    Simulate a model generation/repair attempt.
    Each round has a higher probability of success, modeling
    the improvement that execution feedback provides.
    """
    np.random.seed(attempt * 17 + 42 + seed_offset)
    repair_bonus = attempt * 0.18
    success_prob = min(0.95, base_pass_rate + repair_bonus)
    success = np.random.random() < success_prob

    if success:
        feedback = "All tests passed."
    else:
        errors = [
            "AssertionError: expected 2.5, got 2",
            "TypeError: comparison not supported between str and int",
            "IndexError: list index out of range",
            "ValueError: median of empty sequence",
        ]
        feedback = f"Test failed: {errors[attempt % len(errors)]}"

    return success, feedback

Out[11]:

Console

Self-Repair Loop Simulation
========================================

Problem 1:
  Round 1: FAIL | Test failed: AssertionError: expected 2.5, got 2
  Round 2: FAIL | Test failed: TypeError: comparison not supported betwee
  Round 3: PASS | All tests passed.

Problem 2:
  Round 1: FAIL | Test failed: AssertionError: expected 2.5, got 2
  Round 2: FAIL | Test failed: TypeError: comparison not supported betwee
  Round 3: PASS | All tests passed.

Problem 3:
  Round 1: FAIL | Test failed: AssertionError: expected 2.5, got 2
  Round 2: PASS | All tests passed.

Out[12]:

Visualization

Line chart rising from 42 percent solved initially to about 74 percent after three repair attempts, with diminishing gains at each stage. — A stylized cohort model shows cumulative solve rate across an initial generation and three repair attempts. The first feedback-guided repair produces the largest gain, while later attempts resolve progressively smaller shares of the remaining problems.

The stylized example illustrates the pattern reported in self-repair systems: the first repair can fix many obvious failures, while later attempts operate on a progressively harder residual set. The exact gains depend on the model, tests, and feedback quality; the plotted rates are illustrative rather than benchmark measurements.

Docstring Quality and Generation SuccessLink Copied

The quality of the natural language specification has a measurable effect on generation quality. We can simulate this by scoring docstrings for specificity and showing the relationship between docstring quality scores and generation success rates.

In[13]:

Code

def score_docstring_quality(docstring):
    """
    Score a docstring's specificity and completeness.
    Returns a value in [0, 1], where higher means more specific.
    """
    score = 0.0
    max_score = 6.0

    words = docstring.lower().split()
    n_words = len(words)

    # Length bonus
    if n_words > 10:
        score += 1.0
    elif n_words > 5:
        score += 0.5

    # Mentions return behavior
    if any(w in words for w in ["return", "returns", "output", "result"]):
        score += 1.0

    # Mentions edge cases
    if any(
        w in words
        for w in ["empty", "none", "null", "zero", "negative", "duplicate"]
    ):
        score += 1.0

    # Mentions types or constraints
    if any(
        w in words
        for w in ["list", "string", "int", "float", "integer", "sorted"]
    ):
        score += 1.0

    # Includes example phrasing
    if any(w in words for w in ["example", "e.g.", "given", "for"]):
        score += 1.0

    # Specifies boundary conditions
    if any(
        w in words for w in ["even", "odd", "average", "both", "two", "middle"]
    ):
        score += 1.0

    return score / max_score

Out[14]:

Visualization

Scatter plot of 40 simulated problems with a noisy upward trend between docstring quality score and generation success rate. — A simulated set of code-generation problems shows a positive relationship between docstring quality and success rate, with realistic variation around the trend. Clearer specifications improve the odds of success but do not guarantee it.

The positive trend supports the practical intuition that more specific docstrings help, while the scatter shows that specification quality is not the only factor. Clear edge cases, return behavior, and type constraints reduce ambiguity and can lower iteration costs downstream.

Key ParametersLink Copied

The key parameters for code generation pipelines are:

temperature: Controls output diversity. Values of 0.2 to 0.5 work well for code, balancing correctness with some variation for best-of-N selection.
top_p (nucleus sampling threshold): Typical values of 0.90 to 0.95 keep high-probability tokens while allowing occasional exploration.
n (candidates per problem): More candidates dramatically improve pass@k. Even n=5 roughly doubles solve rates on moderately difficult problems compared to n=1.
max_tokens: Set large enough to avoid truncating function bodies. For typical function implementations, 256 to 512 tokens is usually sufficient.
repair iterations: One or two repair rounds provide most of the gain; three or more have diminishing returns.