Code Execution › Code Execution

When a language model writes code, producing syntactically correct text is only half the task. The real measure of a code-writing system is whether the code runs and produces correct results. Code execution closes the loop between generation and verification, turning an LLM from a text generator into an agent that can test, observe, and refine its own outputs.

The history of programming environments offers a useful frame for thinking about why execution matters so much. Before interactive computing, programs were submitted as batch jobs on punch cards, ran overnight, and returned results (or errors) the next morning. The feedback cycle was so long that programmers invested enormous effort in desk-checking code before submitting it. Interactive terminals shortened that loop to seconds, dramatically changing how developers worked. Integrated development environments shortened it further, giving inline type checking and lint feedback without even running the code. Each reduction in feedback latency changed speed and the entire cognitive process: programmers became more willing to experiment, more willing to try partial solutions and iterate, because the cost of being wrong dropped.

Language models integrated with execution environments follow the same pattern. An LLM that generates code in a single shot, with no ability to observe what the code does, is equivalent to the batch-job programmer submitting punch cards. An LLM embedded in a generate-execute-refine loop is more like the developer at an interactive terminal, trying something, watching what happens, and adjusting accordingly. The quality of the final output depends on the model's code generation ability and on how tightly the loop is closed.

In previous chapters, we covered how models are trained on code, how they understand and complete code, and how they generate full functions from docstrings or tests. But all of those capabilities produce static text. Code execution adds a dynamic feedback layer: the generated code runs in a real environment, produces observable output, and that output gets fed back into the system for evaluation or further refinement. This shift from generation-only to generation-with-execution is one of the defining characteristics of modern AI coding assistants and autonomous coding agents.

This chapter covers the four pillars of code execution in AI systems: sandboxed execution environments that safely run untrusted code, execution feedback mechanisms that capture and interpret runtime results, iterative refinement loops that use feedback to improve generated code, and execution safety practices that protect users and infrastructure from malicious or accidental harm.