Code Execution › Iterative Refinement
Language AI HandbookAll chapters Code Execution › Iterative RefinementOne section · Sections in this chapter4 / 9 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Execution feedback by itself is useful, but the real power comes from using that feedback in a loop. Iterative refinement is the process of generating code, executing it, observing the result, and generating an improved version, repeating until the code passes or a maximum iteration count is reached.
The Generate-Execute-Refine Loop
The basic structure of iterative refinement is a while loop with three components:
- Generate: Prompt the LLM to produce code (on the first iteration) or to fix code (on subsequent iterations).
- Execute: Run the generated code in a sandbox and capture the feedback.
- Evaluate: Check if the feedback indicates success (exit code 0, all tests passing). If yes, return the code. If no, loop again with the feedback added to the prompt.
This structure appears simple, but getting it right in practice requires attention to how the repair prompt is constructed, how much context to include, and what to do when the loop does not converge. A key design choice is whether to include the full conversation history in each repair prompt or to start fresh each time. Including history gives the model more context about what has already been tried, but it also grows the prompt with each iteration, eventually running into context limits. A common approach is to include the original task, the current code, and the most recent execution feedback, dropping intermediate iterations once the context grows large.
def build_repair_prompt(
task_description: str,
current_code: str,
error_feedback: str,
iteration: int,
) -> str:
"""
Build a prompt asking an LLM to fix code given execution feedback.
"""
fence = chr(96) * 3
prompt = (
"You are a Python expert. Fix the code below so it passes execution.\n\n"
f"Task: {task_description}\n\n"
f"Attempt {iteration} - Current code:\n"
f"{fence}python\n"
f"{current_code}\n"
f"{fence}\n\n"
f"Execution feedback:\n{error_feedback}\n\n"
"Provide ONLY the corrected Python code, no explanations:\n"
f"{fence}python\n"
)
return prompt
def iterative_refine(
task_description: str,
initial_code: str,
tests: str,
llm_fn, # callable: prompt -> code string
max_iterations: int = 5,
) -> dict:
"""
Run the generate-execute-refine loop until tests pass or max iterations reached.
"""
code = initial_code
history = []
for iteration in range(1, max_iterations + 1):
test_result = run_tests_against_code(code, tests)
record = {
"iteration": iteration,
"code": code,
"passed": test_result["passed"],
"failed": test_result["failed"],
"total": test_result["total"],
"success": test_result["failed"] == 0 and test_result["total"] > 0,
}
history.append(record)
if record["success"]:
break
if iteration == max_iterations:
break
# Build feedback string for the LLM
feedback_lines = [
f"{test_result['passed']}/{test_result['total']} tests passed."
]
for detail in test_result["details"][:3]: # limit to first 3 failures
feedback_lines.append(
f"FAIL: {detail['test']}: {detail['message'][:200]}"
)
feedback = "\n".join(feedback_lines)
# Ask LLM to fix the code
prompt = build_repair_prompt(
task_description, code, feedback, iteration
)
code = llm_fn(prompt)
return {"final_code": code, "history": history}This loop structure is the core of many autonomous coding agents. The model does not need to get the code right on the first attempt; it needs to get it right eventually, using execution feedback as a signal to improve. This more closely mirrors how human programmers work than single-shot generation does.
Why Iterative Refinement Works: A Mental Model
To understand why iterative refinement is so effective, consider the space of all possible programs that could solve a given task. In single-shot generation, the model must pick a point in this space that lands in the correct region on the first try. The probability of success depends on how precisely the task is described, how well the model's training covers the required patterns, and how complex the task is.
With iterative refinement, each execution provides information about where in the program space the current attempt falls. A failing test tells you that the current point is outside the correct region and gives a constraint (the failing assertion) that defines the boundary. Each iteration takes a step toward the correct region, guided by these constraints. The process is analogous to gradient descent in the space of programs: not a smooth gradient (since programs are discrete and execution is non-differentiable), but a directed search guided by structured feedback.
This framing also explains when iterative refinement fails. If the error messages are uninformative (the model cannot determine the direction to move) or if the correct region of program space is very small (the task is underspecified), convergence may not occur. It also explains why test coverage quality matters: more tests provide more constraints, more precisely defining the correct region and giving clearer navigation signals.
Convergence and Failure Modes
Iterative refinement is not guaranteed to converge. Several failure modes occur in practice:
Cycling: The model oscillates between two broken implementations, alternating between two different bugs. Each repair introduces the other bug. Detecting cycles requires hashing previous code states and breaking out of the loop when a duplicate is seen.
Regression: A repair that fixes one failing test breaks a previously passing test. This is especially common when the model is not shown all test failures simultaneously, only the first one. A well-designed harness shows all failures at once and tracks pass counts across iterations to detect regression.
Context window saturation: As the loop runs, the prompt accumulates feedback from multiple iterations. If each iteration adds a long traceback, the context window fills quickly. Strategies to manage this include summarizing past iterations, truncating feedback, or periodically resetting to only the original task plus the current best code.
Task underspecification: When the task description is ambiguous, the model may produce code that passes the tests but does not do what the user intended. Tests that do not fully specify the desired behavior lead to "passing but wrong" outcomes. This is a fundamental limitation of test-based feedback: the tests are a proxy for correctness, not a complete specification.
Scaffolding errors: In some cases the generated code is correct but the test harness is broken, or the test expectations are wrong. The model, seeing test failures, tries to change its correct implementation to match the broken tests. Distinguishing "the code is wrong" from "the test is wrong" requires either human oversight or formal verification.
def detect_cycle(history: list, lookback: int = 3) -> bool:
"""
Check if the last `lookback` code states form a cycle.
Returns True if a repeated code string is found.
"""
if len(history) < lookback:
return False
recent_codes = [h["code"] for h in history[-lookback:]]
return len(set(recent_codes)) < len(recent_codes)
def track_regression(history: list) -> list:
"""
Identify iterations where the pass count dropped from the previous iteration.
Returns a list of (iteration, pass_before, pass_after) tuples.
"""
regressions = []
for i in range(1, len(history)):
prev = history[i - 1]["passed"]
curr = history[i]["passed"]
if curr < prev:
regressions.append((history[i]["iteration"], prev, curr))
return regressions
# Simulate a refinement history with a cycle
simulated_history = [
{
"iteration": 1,
"code": "def f(x): return x",
"passed": 2,
"failed": 3,
"total": 5,
"success": False,
},
{
"iteration": 2,
"code": "def f(x): return x + 1",
"passed": 4,
"failed": 1,
"total": 5,
"success": False,
},
{
"iteration": 3,
"code": "def f(x): return x",
"passed": 2,
"failed": 3,
"total": 5,
"success": False,
},
{
"iteration": 4,
"code": "def f(x): return x + 1",
"passed": 4,
"failed": 1,
"total": 5,
"success": False,
},
]
cycle_detected = detect_cycle(simulated_history, lookback=4)
regressions = track_regression(simulated_history)Cycle detected: True Regressions found: 1 Iteration 3: passed dropped from 4 to 2 Iteration history: Iter 1: 2/5 pass Iter 2: 4/5 pass Iter 3: 2/5 pass Iter 4: 4/5 pass
Detecting cycles and regressions early and breaking the loop (or changing the repair strategy) avoids wasting compute on iterations that are unlikely to converge.
In practice, many bugs that iterative refinement can fix are resolved within the first two or three attempts. The cumulative pass rate tends to rise steeply in early iterations and flatten as the remaining failures become harder. The following illustrative simulation makes that pattern concrete without claiming benchmark measurements.
Self-Debugging Agents
The iterative refinement loop described above is the foundation of self-debugging agents. These are LLM-based systems that autonomously fix their own code without human intervention, using execution feedback as their only signal.
Research on self-debugging has shown that models with access to execution feedback significantly outperform models that generate code in a single shot, even when both use the same base model. The gain comes from two sources: the model can observe the actual runtime behavior of its code (catching bugs that static reasoning would miss), and the additional iterations give the model multiple chances to succeed (acting like a form of self-consistency sampling).
The key insight is that LLMs are better at recognizing a bug given an error message and code than they are at writing perfectly correct code from scratch. This asymmetry comes from the nature of language model training: the pretraining data contains vastly more examples of "here is an error and here is how to fix it" (in the form of code reviews, Stack Overflow answers, commit messages, and debugging sessions) than examples of "write this complex function perfectly on the first try." Execution feedback plays to this strength by converting the hard single-shot generation task into an easier recognition-and-repair task.
Multi-Step Code Agents
For complex programming tasks, a single generate-execute-refine loop over one function may not be enough. Multi-step code agents break the task into subtasks, execute each one, and use the results to inform subsequent steps.
Consider a task like "scrape a web page, extract all prices, and write a CSV file." A multi-step agent might:
- Generate and execute code to fetch the web page.
- Observe the raw HTML output and use it to generate code for parsing.
- Generate and execute the parsing code on the real HTML.
- Use the parsed results to generate and execute the CSV-writing code.
- Verify the output file exists and contains expected data.
Each step produces intermediate outputs that serve as grounding for the next step. This grounded, sequential approach is more reliable than attempting to write the entire pipeline in one shot, because each step verifies its own output before the next step begins. The execution environment acts as a shared memory between steps: the agent can inspect variables, print intermediate data structures, and observe file system state to validate that each step achieved what it intended.
The pass@k Metric and Parallel Sampling
An alternative to sequential iterative refinement is parallel sampling: generating many candidate solutions simultaneously and returning the best one (or the first one that passes). The pass@k metric, which we will cover in detail in the next chapter, formalizes this: it measures the probability that at least one of independently generated solutions solves the problem.
Parallel sampling and sequential refinement represent different points on a compute-versus-latency tradeoff. Parallel sampling uses more compute overall (since all samples are generated and executed) but can be parallelized, leading to lower wall-clock latency if sufficient compute is available. Sequential refinement uses less compute (the loop stops as soon as a solution passes) but is inherently sequential and thus slower in terms of wall-clock time. Systems like AlphaCode generate hundreds of candidates in parallel, which is only feasible because LLM inference can be batched and execution can be distributed across many containers simultaneously.
For interactive developer tools, sequential refinement is usually preferred because it is cheaper and produces an explanation of what went wrong at each step. For benchmark evaluation and offline quality optimization, parallel sampling with large gives the best final accuracy at the cost of more compute.