Code Execution › Execution Feedback
Language AI HandbookAll chapters Code Execution › Execution FeedbackOne section · Sections in this chapter3 / 9 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Sandboxing allows code to run safely. Execution feedback is what makes that execution useful for improving the code. The output produced by execution, whether a successful result, an error message, a stack trace, or a test report, is information that can be fed back into the LLM to guide the next generation step.
The fundamental insight behind execution feedback as a learning signal is that code errors are structured and informative. Unlike a generic "your code is wrong" message, a Python TypeError: unsupported operand type(s) for +: 'int' and 'str' tells you exactly what went wrong, where, and what the types involved were. A failing assertion AssertionError: assert sorted(result) == [0, 1], but got [1, 2] tells you that the function returned wrong values and what the correct values should be. This structural richness is what allows an LLM to act on execution feedback rather than just knowing that something failed.
Capturing Execution Output
Execution feedback begins by capturing of everything the code produces. A reliable execution engine captures:
- Standard output (stdout): The text printed by the program, typically the intended result.
- Standard error (stderr): Error messages and warnings. In Python, uncaught exceptions print their tracebacks here.
- Exit code: Zero for success, non-zero for failure. This provides an immediate signal about whether execution completed normally.
- Execution time: How long the code took to run, useful for diagnosing performance issues.
- Exception type and message: When an exception is raised, the type (
TypeError,KeyError, etc.) and message provide structured information about what went wrong. - Stack trace: The full traceback showing which line caused the error and the call chain leading to it.
import subprocess
import textwrap
import time
def execute_code(code: str, timeout: int = 10) -> dict:
"""
Execute Python code in a subprocess and capture all output.
Returns a dict with stdout, stderr, exit_code, elapsed_time.
"""
# Write code to a temp file to avoid shell injection issues
import os
import tempfile
with tempfile.NamedTemporaryFile(suffix=".py", mode="w", delete=False) as f:
f.write(code)
tmp_path = f.name
start = time.monotonic()
try:
result = subprocess.run(
["python3", tmp_path],
capture_output=True,
text=True,
timeout=timeout,
)
elapsed = time.monotonic() - start
return {
"stdout": result.stdout,
"stderr": result.stderr,
"exit_code": result.returncode,
"elapsed": elapsed,
"timed_out": False,
}
except subprocess.TimeoutExpired:
elapsed = time.monotonic() - start
return {
"stdout": "",
"stderr": f"Execution timed out after {timeout} seconds.",
"exit_code": -1,
"elapsed": elapsed,
"timed_out": True,
}
finally:
os.unlink(tmp_path)
# Test with a simple example
sample_code = textwrap.dedent("""
def factorial(n):
if n <= 1:
return 1
return n * factorial(n - 1)
print(factorial(5))
print(factorial(10))
""")
result = execute_code(sample_code)Exit code: 0 Stdout: 120 3628800 Elapsed: 0.027s Timed out: False
The execution harness captures output at the process level, which means it works regardless of what language or runtime the generated code uses. The same harness can run Python, Node.js, or compiled binaries by changing the command passed to subprocess.run. Note that we write to a temp file rather than passing code via stdin or shell arguments: this avoids shell injection vulnerabilities (where specially crafted code strings could escape the intended command) and handles code that spans multiple lines cleanly.
Interpreting Error Output
When execution fails, the raw error message is valuable but often too verbose or low-level for an LLM to act on effectively. Execution systems often include a parsing layer that extracts structured information from error output.
For Python exceptions, the standard format includes the traceback (showing the call stack and file and line information), the exception class, and the message. A parser can extract these components. The innermost frame of the traceback is usually the most relevant, because it shows the exact line that triggered the error. The exception type is important too: a TypeError calls for different repair strategies than a KeyError or an IndexError.
import re
def parse_python_error(stderr: str) -> dict:
"""
Parse a Python traceback to extract structured error information.
Returns the exception type, message, and the most relevant line.
"""
lines = stderr.strip().split("\n")
error_info = {
"raw": stderr,
"exception_type": None,
"message": None,
"line_number": None,
"line_code": None,
}
# Find the last "File ..., line N" entry (the innermost frame)
file_line_pattern = re.compile(r'File "(.+)", line (\d+), in (.+)')
last_frame = None
for line in lines:
match = file_line_pattern.search(line)
if match:
last_frame = match
if last_frame:
error_info["line_number"] = int(last_frame.group(2))
# Extract the exception type and message from the last line
if lines:
last_line = lines[-1]
colon_idx = last_line.find(":")
if colon_idx != -1:
error_info["exception_type"] = last_line[:colon_idx].strip()
error_info["message"] = last_line[colon_idx + 1 :].strip()
else:
error_info["exception_type"] = last_line.strip()
return error_info
# Test with a failing example
failing_code = textwrap.dedent("""
def add(a, b):
return a + b
result = add(10, "twenty") # TypeError
print(result)
""")
failed_result = execute_code(failing_code)
parsed_error = parse_python_error(failed_result["stderr"])Exit code: 1
Raw stderr:
Traceback (most recent call last):
File "/var/folders/lz/vn3ps0t51nv5q2g7q4kppt1r0000gn/T/tmpr27cf21s.py", line 5, in <module>
result = add(10, "twenty") # TypeError
^^^^^^^^^^^^^^^^^
File "/var/folders/lz/vn3ps0t51nv5q2g7q4kppt1r0000gn/T/tmpr27cf21s.py", line 3, in add
return a + b
~~^~~
TypeError: unsupported operand type(s) for +: 'int' and 'str'
Parsed error:
Type: TypeError
Message: unsupported operand type(s) for +: 'int' and 'str'
Line: 3Parsing the exception type and message allows downstream components (like an LLM prompt builder) to highlight the most relevant information without flooding the context window with a long traceback. For a model with a 128K token context window this might seem unimportant, but in practice shorter, more focused prompts tend to produce better repair results than prompts padded with noise.
Test-Based Feedback
In agentic coding workflows, the most valuable feedback often comes not from the program's output but from a test suite. When a model generates code to implement a function, running the associated unit tests provides precise, structured feedback: which tests pass, which fail, and why.
Test-based feedback is especially powerful because it encodes the intended behavior of the code independently of the implementation. A failing test tells the model exactly what property its code violates, with a precise assertion that can be inserted directly into a repair prompt. Consider the difference between these two pieces of feedback for a sort_list function:
Feedback type 1 (raw execution output): [3, 1, 2]
Feedback type 2 (test failure): AssertionError: Lists differ: [3, 1, 2] != [1, 2, 3]
The second version tells the model what the code produced and what it should have produced. The model does not need to infer what "correct" means; the test tells it directly. This is why test-based repair consistently outperforms repair from raw output in self-debugging benchmarks.
def run_tests_against_code(implementation_code: str, test_code: str) -> dict:
"""
Execute implementation code and then run tests against it.
Returns pass/fail counts and per-test details.
"""
combined = implementation_code + "\n\n" + test_code + "\n\n"
combined += textwrap.dedent("""
import unittest
loader = unittest.TestLoader()
suite = loader.loadTestsFromTestCase(TestSolution)
runner = unittest.TextTestRunner(verbosity=2, stream=open('/dev/null', 'w'))
result = runner.run(suite)
total = result.testsRun
failed = len(result.failures) + len(result.errors)
passed = total - failed
print(f"TESTS:{total}:{passed}:{failed}")
for f in result.failures:
print(f"FAIL:{f[0].id()}:{f[1][:200]}")
for e in result.errors:
print(f"ERROR:{e[0].id()}:{e[1][:200]}")
""")
exec_result = execute_code(combined)
parsed = {"total": 0, "passed": 0, "failed": 0, "details": []}
for line in exec_result["stdout"].split("\n"):
if line.startswith("TESTS:"):
parts = line.split(":")
parsed["total"] = int(parts[1])
parsed["passed"] = int(parts[2])
parsed["failed"] = int(parts[3])
elif line.startswith("FAIL:") or line.startswith("ERROR:"):
parts = line.split(":", 2)
parsed["details"].append(
{
"test": parts[1],
"message": parts[2] if len(parts) > 2 else "",
}
)
return parsed
# Example: a model-generated solution to a simple problem
implementation = textwrap.dedent("""
def is_palindrome(s):
s = s.lower().replace(" ", "")
return s == s[::-1]
""")
tests = textwrap.dedent("""
import unittest
class TestSolution(unittest.TestCase):
def test_simple(self):
self.assertTrue(is_palindrome("racecar"))
def test_spaces(self):
self.assertTrue(is_palindrome("A man a plan a canal Panama"))
def test_false(self):
self.assertFalse(is_palindrome("hello"))
def test_empty(self):
self.assertTrue(is_palindrome(""))
""")
test_results = run_tests_against_code(implementation, tests)Test results: Total: 4 Passed: 4 Failed: 0
Test-based feedback achieves a good balance between signal richness and noise reduction. A passing test suite gives the model a strong signal to stop iterating. A failing test identifies the specific property that needs to be repaired, which is more actionable than a raw exception message.
Execution Feedback Formatting
Once raw execution output has been captured and optionally parsed, it needs to be formatted into a prompt that an LLM can act on. The format matters: too much raw output floods the context window; too little leaves out information the model needs.
The goal of feedback formatting is not to show the LLM everything but to show it the right things in the right order. A well-constructed feedback prompt places the error information adjacent to the offending code rather than at the end of a long dump. This mirrors how human developers read error messages: they jump to the relevant line, look at the surrounding code, and diagnose the issue. By formatting the feedback to support this pattern, you make the repair task easier for the model.
Effective feedback formatting follows these principles:
- Include the relevant portion of the original code so the model can see what it wrote.
- Place the error message near the code line that caused it, not at the end of a long traceback.
- Truncate long outputs at a reasonable character limit, marking truncations explicitly.
- Summarize test results with counts rather than full test output when many tests pass.
- Label the feedback clearly (e.g., "Execution Result:", "Error:", "Test Failure:") so the model understands the structure.
Different feedback types provide different amounts of actionable information to the model. Raw exception messages identify the error type but not the violated specification. Full stack traces add location context but can be noisy. Test-based feedback is the most specific, telling the model exactly which behavior is wrong. The chart below uses illustrative simulated rates to show that ordering; it is not a benchmark result.
Feedback as Grounding
There is a deeper point about why execution feedback works so well: it provides grounding. When a model generates code purely from a natural language description, it relies entirely on its learned representation of what that description means. When it receives execution feedback, it sees the actual runtime behavior of its code: the values that variables held, the exception that was raised, the assertion that failed. This grounds the model's next generation step in observable reality rather than in its statistical expectations.
This parallels a phenomenon in cognitive science called "situated cognition": humans solve problems more effectively when they can interact with a real environment than when they reason abstractly. The programmer who runs their code and watches the output learns faster than the one who only traces through it mentally. Execution feedback gives LLMs a similar advantage.