Code Generation › Limitations and Practical Implications

Code generation models are powerful but have clear failure modes that become apparent with real-world use. The most common is specification ambiguity: when the docstring or tests do not fully specify the function's behavior, the model makes choices based on training distribution priors that may not match the developer's intent. This is not a failure of the model; it is a fundamental limitation of underspecified input.

A more subtle limitation is distributional mismatch. Models are trained on publicly available code, which has a particular distribution of programming patterns, library choices, and idiomatic styles. Internal codebases often differ substantially from public code. Functions that interact with proprietary APIs, use internal utility libraries, or follow company-specific conventions will be poorly generated by models that have never seen those patterns. This is why context-augmented generation (providing examples from the target codebase in the model's context) often improves quality significantly on internal codebases.

Long-range dependency is another persistent challenge. Simple functions with local logic are generated well. Functions that must maintain consistency with complex state, follow multi-step protocols, or interact with global configuration are harder. The model's effective context window limits how much of the surrounding codebase it can reason about simultaneously.

There is also a subtler limitation around logical consistency in longer code segments. When generating a short function, the model can maintain coherence between its opening and closing lines easily. As the function grows longer or spans multiple interacting components, the probability of internal inconsistencies increases. The model may initialize a variable in a way that conflicts with how it uses it fifty lines later, introduce a loop that accumulates values incorrectly, or return a result that does not account for a conditional branch established earlier in the function. These errors are not random; they tend to occur at the boundaries of the model's "working memory" within a single generation pass.

Deploying code generation in production introduces security concerns that are absent from benchmarking settings. Models can generate code with security vulnerabilities: SQL injection vulnerabilities, path traversal issues, inadequate input validation, weak cryptographic choices. Benchmark evaluation measures functional correctness on provided tests, which rarely include security test cases.

Several studies have found that code generated by large language models has measurably higher rates of certain vulnerability patterns than human-written code reviewed by security teams. This does not mean generated code should be avoided in production; it means it should be treated with the same (or higher) scrutiny as code from an untrusted source. Automated static analysis, security-focused linting, and code review remain essential.

The vulnerability patterns most commonly found in generated code are not exotic. They include using string formatting to construct SQL queries (enabling injection), not validating file paths before opening them, using deprecated or weak cryptographic functions, and not handling authentication errors securely. These patterns appear in publicly available code and are therefore represented in training data, which means the model learns to reproduce them in contexts where they are inappropriate.

The mitigation strategy is to treat security properties as part of the specification, not as implicit requirements. Prompts that explicitly state security requirements ("use parameterized queries, not string formatting," "validate and sanitize all file paths") produce more secure code. Static analysis tools that specifically check for security anti-patterns should be part of any production pipeline that incorporates generated code.

HumanEval and MBPP, the dominant benchmarks for code generation, consist of relatively short, self-contained function implementations with clear specifications. This is very different from the code that developers need generated in practice: functions that interact with frameworks, handle complex data schemas, implement multi-step protocols, or require domain-specific knowledge.

Models that achieve high pass@1 rates on HumanEval may perform much worse on internal task-specific benchmarks. When adopting code generation tools, running evaluations on a sample of your own tasks provides a much more reliable estimate of actual utility than benchmark scores. The code evaluation chapter covers benchmark design and interpretation in more detail.

The benchmark-reality gap also manifests in the distribution of difficulty. HumanEval problems have a specific difficulty profile, dominated by medium-difficulty algorithmic problems that a competent programmer could implement in a few minutes. Real codebases include a different distribution: many trivially easy functions (getters, setters, simple data transformations) that models handle well, some moderately complex functions similar to HumanEval problems, and a non-trivial fraction of highly domain-specific functions that are much harder than anything in HumanEval. Benchmark scores disproportionately reflect performance on the medium tier. This means that benchmark scores underestimate performance on easy production tasks and overestimate it on hard ones.

The appropriate level of human oversight for generated code is a design choice that depends on the deployment context, the criticality of the code, and the quality of the testing infrastructure. Several patterns have emerged from engineering teams that deploy code generation at scale.

In low-stakes contexts with broad test coverage, generated code can be accepted automatically if all tests pass. This is common for generated tests themselves, for simple utility functions with clear specifications, and for boilerplate code in well-tested frameworks. In medium-stakes contexts, generated code is reviewed by the developer who requested it. The review is typically much faster than writing from scratch because the reviewer is verifying rather than creating.

In high-stakes contexts (security-critical paths, financial calculations, safety-relevant logic), generated code is reviewed with the same rigor as human-written code, possibly with additional scrutiny given the known limitations of generated code in these domains. Some teams treat generated code as always requiring a second reviewer regardless of apparent simplicity.

The key insight is that the appropriate oversight level is not a fixed property of code generation as a technology; it depends on the specific application context and the quality of automated verification available. Treating all generated code with maximum scrutiny is wasteful. Treating all generated code as equivalent to thoroughly reviewed human code is risky. Finding the right calibration for each use case is itself an engineering problem worth solving deliberately.