Code Generation › Summary

Code generation bridges the gap between natural language specification and executable implementation. Two primary formulations capture most practical use cases: docstring-to-code generation, where a natural language function description drives generation, and test-to-code generation, where unit tests serve as an executable specification.

Generation quality depends on both the model and the generation strategy. Key takeaways:

• Sampling temperature matters: use 0.2 to 0.5 for code generation rather than the higher values used for creative text
• Best-of-N selection dramatically improves pass@k rates; even generating five candidates instead of one roughly doubles solve rates on moderate problems
• Self-repair loops that incorporate execution feedback further improve correctness; one or two repair rounds provide most of the benefit
• Input specification quality is a major driver of output quality; clearer docstrings with edge case descriptions produce better code
• Benchmark scores are not the full picture: evaluate on representative samples of your own tasks
• Security and distributional mismatch are the most practically important limitations in production deployment
• Context management and prompt engineering are as important as model selection for production code generation quality

Code generation methods continue to change rapidly. Models are becoming more capable, context windows are growing, and tool-augmented generation (where the model can execute code, look up APIs, and inspect types during generation) is beginning to blur the line between single-pass generation and the agentic code generation systems of the near future. The foundations covered here, specification quality, sampling strategies, verification, and limitation awareness, remain relevant regardless of how the models themselves continue to improve.

The Code Execution chapter covers the execution environment and feedback mechanisms that make self-repair loops possible. The Code Evaluation chapter provides a deeper treatment of pass@k, HumanEval, and the methodological challenges of measuring code generation quality rigorously.