Code Generation › Test-to-Code Generation

Test-to-code generation replaces natural language descriptions with executable unit tests as the specification. This section explains why tests make better specifications than docstrings, what the model must do to generalize beyond the provided tests, and how this formulation fits into test-driven development workflows.

Test-to-code generation treats unit tests as the complete specification. The model receives one or more test cases and must generate a function implementation that passes all of them. This formulation has a key advantage over docstring-to-code: the tests are executable and unambiguous. There is no question about whether the generated code is "correct" relative to a vague description. Either the tests pass or they do not.

A typical test-to-code input looks like this:

Given these tests, the model must generate a find_median function. The tests encode the edge case behavior explicitly. Empty lists might still be unspecified, but many common ambiguities are resolved. The even-length case is handled (return the average of the two middle values, yielding 2.5 for [1,2,3,4]). The single-element case is specified (return that element). The negative-number case is covered.

What makes tests particularly powerful as specifications is that they are machine-checkable. There is no room for the kind of interpretive flexibility that natural language allows. If the model generates a function that returns 2 for [1,2,3,4] instead of 2.5, the test fails immediately and the error is reported with full precision. This tight feedback loop is what makes test-to-code generation so well suited to automated pipelines.

The challenge in test-to-code is that tests are a subset of the function's full behavior. A model that generates code passing the provided tests may still fail on unseen inputs. This is not a flaw in the approach; it reflects a fundamental property of testing. No finite set of tests can fully specify a function over all possible inputs.

The model must generalize from the provided tests to the underlying intent. A model that memorizes exactly what the tests check will produce brittle code. A model that infers the general pattern (median computation with proper handling of even-length lists) will produce robust code.

This generalization requirement creates an interesting asymmetry. A sufficiently adversarial model could generate a function that special-cases the exact test inputs and returns the correct answer for each case while doing something completely wrong for all other inputs. In practice, this does not happen because language models are not adversarial reasoners: they learn to generate code that reflects the general pattern implied by the tests, not code that memorizes specific input-output pairs. But the fact that this is technically possible illustrates why test-to-code evaluation must always include hidden tests that the model never sees.

The quality of test-to-code generation also depends heavily on how informative the provided tests are. A test suite with good coverage of edge cases (empty inputs, boundary values, type edge cases) gives the model much more information than a test suite that only covers the happy path. Models presented with tests that cover the specified behavior tend to generate more robust implementations than models given minimal tests, even when the minimal tests are technically sufficient to define the function's behavior.

One underappreciated aspect of test-to-code generation is that tests can encode behavior and intent. Consider two different implementations of find_median that both pass the four tests above. One uses sorted() and index arithmetic. Another uses heapq.nsmallest(). Both are correct for the given tests. But a well-written model will prefer the sorted() approach because the tests themselves use small sorted lists and integer comparisons, which pattern-matches strongly to the sorting-based implementation in training data.

This implicit alignment between test style and implementation style is an underexplored dimension of test-to-code quality. Tests written in a way that reflects the intended algorithmic approach tend to produce better-aligned implementations than tests that are implementation-agnostic. This matters in practice when developers write tests with a specific implementation strategy in mind and want the generated code to reflect that strategy.

Test-to-code generation aligns naturally with test-driven development (TDD), where tests are written before implementation. In a TDD workflow, the developer writes tests that specify desired behavior, and then a code generation model (or the developer themselves) implements the function. The model's output is immediately verifiable.

This integration changes how developers interact with code generation tools. Instead of prompting with a description and hoping the output is correct, the developer writes tests first, then generates an implementation, then runs the tests to verify. The feedback loop is tighter and the verification is automatic.

TDD-integrated code generation also changes the economics of code review. When generated code is paired with tests that the reviewer can run, reviewing becomes much faster than evaluating unconstrained generated code. The reviewer can verify behavior empirically rather than reasoning about correctness from first principles. Several engineering teams have reported that this combination, writing tests manually then generating implementations automatically, dramatically speeds up development for well-understood functionality while preserving human judgment for the test design itself, which requires deeper domain understanding.

The limitation of TDD-integrated generation is that writing good tests is itself a non-trivial skill. Tests that adequately specify complex behavior require the developer to think carefully about edge cases, invariants, and failure modes. In some sense, the work is not eliminated but redistributed: from writing implementations to writing tests. Whether this is preferable depends on the domain and the complexity of the code.