Code Execution › Summary

This chapter covered the four pillars of code execution in AI systems:

• Sandboxed execution isolates generated code using containers, VMs, or Wasm to prevent damage to the host system. Resource limits (CPU time, memory, process count) protect against runaway code. The choice between Docker, gVisor, Firecracker, and WebAssembly involves tradeoffs between startup latency, isolation strength, and ecosystem compatibility.
• Execution feedback captures stdout, stderr, exit codes, and test results. Parsing this output into structured form (exception type, failing test names, expected versus actual values) makes it actionable for LLM repair prompts. Test-based feedback is the most powerful type because it encodes the intended behavior of the code, not just that something went wrong.
• Iterative refinement uses execution feedback in a generate-execute-refine loop. The model repairs its own code across multiple iterations, using test failures as the primary signal. Cycle detection and regression tracking prevent the loop from spinning without progress. Most fixable bugs resolve within two or three iterations; the pass rate curve flattens for the hardest problems.
• Execution safety addresses threats from adversarial inputs, prompt injection, and accidental harm through a layered architecture: prompt-level safety, static analysis, sandbox isolation, resource limits, and human-in-the-loop checkpoints for irreversible actions. No single layer is sufficient; the combination provides defense in depth.

Together, these mechanisms transform a code-generating language model into an agent that can interact with a real execution environment, observe results, and refine its outputs over time. The resulting systems are more capable and more reliable than static generation alone, but they introduce new engineering challenges around latency, test quality, and security that require careful design to address.