Code Execution › Limitations and Practical Implications

The most fundamental limitation of execution-based feedback is that passing tests does not guarantee correctness. Tests can only verify that a program behaves correctly on the specific inputs they check. LLMs sometimes generate code that "cheats" on tests, returning hardcoded values that happen to match the test cases without implementing the actual algorithm. This is a known failure mode called test-set overfitting or shortcut learning.

A famous example of this from competitive programming benchmarks: early LLM evaluations found that models would occasionally detect the expected output format from the test cases, hard-code the outputs, and pass evaluation without solving the underlying problem. Benchmark designers responded by using private test sets that the model never sees, but this arms race between evaluation design and model behavior is ongoing.

More subtly, even well-designed tests may fail to cover all important edge cases. A function that correctly handles all tested inputs but silently produces wrong results on untested inputs is particularly dangerous in production, because it passes all checks and deploys without warning. Property-based testing (generating random inputs and checking invariants rather than checking specific expected outputs) partially addresses this, but automatically generating good property-based tests for LLM-written code is itself an open research problem.

Running code in a sandboxed container has non-trivial latency. Even with pre-warmed containers, execution overhead adds 50-500 milliseconds per round trip. In an iterative loop with 5 iterations, this adds up to several seconds of execution overhead, on top of the LLM inference time for each generation step. For interactive use cases where users expect responses in under a second, iterative refinement may not be compatible with latency requirements.

Batching and parallelization help. Systems like AlphaCode generate many candidate solutions in parallel (often hundreds) and execute all of them simultaneously, selecting the best-passing solution. This trades sequential iteration depth for breadth, which is more computationally efficient at scale when many GPUs are available for inference. But for individual developer workflows using a cloud API, the per-call cost of generating hundreds of completions is prohibitive.

The practical resolution for most products is to offer iterative refinement for correctness-critical tasks (running tests in CI, solving well-specified algorithmic problems) and to use single-shot generation for interactive tasks (code completion, quick transformations) where the user can correct mistakes manually with lower friction than waiting for multiple refinement rounds.

Code that involves randomness, timing, or external state is difficult to evaluate through execution alone. A function that generates a random shuffle is correct if it produces a valid permutation, but any given execution only checks one particular shuffle. Tests for such functions need careful design (checking properties of the output rather than exact values), which is itself a hard task to automate.

Similarly, code that reads from the web, queries a database, or depends on the current time produces different outputs across executions. Testing such code in a sandbox requires mocking external dependencies, which requires understanding what the code does before you run it. This circular dependency complicates for fully automated execution-based evaluation. Research directions include using LLMs to generate mocks automatically (using the model's understanding of the code to infer what dependencies it needs), but this remains an active research area.

Despite sandboxing and static analysis, execution safety remains an ongoing challenge. New sandbox escape techniques appear regularly, and the cat-and-mouse game between safety systems and adversarial inputs continues. Kernel CVEs, container escape exploits, and WASM side-channel attacks are all discovered periodically, even for systems thought to be secure. The practical implication is that no execution environment should be considered fully secure. Defense in depth, combining multiple layers of protection, is the correct engineering posture, not reliance on any single mechanism.

The emergence of agentic systems that take autonomous actions compounds this challenge. A coding assistant that can only generate code that the user then reviews is much easier to secure than one that can autonomously commit to version control, deploy to staging, and call external APIs. As agents become more capable and autonomous, the attack surface expands, and the consequences of a security failure become more severe.

Despite these limitations, execution-based feedback has substantially changed the capabilities of LLM-powered developer tools. Systems like GitHub Copilot, Cursor, and Devin all use some form of execution feedback, whether running tests in the IDE, checking type errors in the language server protocol, or executing code in an integrated terminal. The shift from static code generation to dynamic, execution-grounded generation is one of the key factors that makes these tools feel qualitatively more capable than previous generations of code suggestion tools.

The pattern we see emerging is a spectrum of autonomy that scales with the reversibility of actions. Reading files, running tests, and compiling code are all fully automated in modern coding assistants. Modifying files requires user confirmation in cautious systems but proceeds automatically in agentic ones. Deploying to production, sending external requests, or modifying databases always require human approval in well-designed systems. This spectrum is not fixed: as trust in the model's judgment increases through track record and as safety tooling improves, the boundary of automation will shift.

The next chapter on Code Evaluation examines how we systematically measure these capabilities, using metrics like pass@k to quantify the probability that at least one of $k$ generated samples passes all tests.