Code Generation › Code Generation

Generating code from natural language descriptions is one of the most commercially impactful applications of language models. A developer describes what they want in plain English, and a model produces a working implementation. What once required parsing a specification document, designing an algorithm, and translating it into syntax now happens in seconds. But the gap between generating code that looks plausible and generating code that works is large, and understanding that gap is what this chapter is about.

Code generation combines language understanding with formal reasoning. Unlike generating a paragraph of prose, generating code requires producing output that is simultaneously syntactically valid, semantically meaningful, and functionally correct. A poem with an awkward phrase is still a poem. A function with a missing return statement is broken. This asymmetry between prose generation and code generation motivates nearly every design decision we cover here.

Building on the foundations from Code LLM Training, Code Understanding, and Code Completion covered earlier in Part XLVII, this chapter focuses on the full-generation setting: producing complete functions, modules, or programs from high-level specifications rather than filling in a single line or block. The prior chapters established how models learn the structure of code and how they operate in interactive completion environments. Here, we ask: given only a description of what code should do, how well can a model produce it?