Code Generation › What Does "Code Generation" Mean?

Code generation is a broad term. In this chapter, we focus on two primary formulations that represent the majority of real-world use cases.

The first is docstring-to-code generation, where the input is a natural language description embedded in or adjacent to a function signature, and the output is the complete function body. This setting is the basis of the HumanEval and MBPP benchmarks and captures the experience of using tools like GitHub Copilot or Claude to implement a function from a description.

The second is test-to-code generation (sometimes called test-driven generation), where one or more unit tests serve as the specification. The model must generate code that passes all tests. This formulation has appealing properties: the specification is precise, the evaluation is automatic, and the generated code can be verified without human judgment.

Both settings share the core challenge: the model must infer the intended behavior from an informal or semi-formal specification, translate that intent into executable logic, and produce output that satisfies constraints the specification may not have made explicit. The challenge is not merely syntactic. Any competent programmer knows Python syntax. The hard part is bridging the semantic gap between a natural language description and the precise computational procedure the description is trying to convey.

Several settings fall outside these formulations. Repository-level code generation, where the model must create entire modules consistent with an existing codebase, is a separate and harder problem. Agentic code generation, where a model iteratively writes, executes, observes, and revises code over many steps, is yet another. This chapter addresses the foundational setting of function-level generation as the cleanest and most thoroughly studied case, with explicit notes about how insights extend to these broader settings.