Test-Time Compute › Iterative Refinement

Multiple sampling generates candidates in parallel and selects among them. Iterative refinement takes a fundamentally different approach: it generates candidates sequentially, using each evaluation to guide the next generation. The model produces an initial answer, evaluates it, and uses the evaluation to produce an improved version. This mirrors the way a human writer drafts and revises: not by writing many independent versions and picking the best, but by progressively improving a single working draft through repeated critique.

The distinction matters because refinement can sometimes reach quality levels that pure parallel sampling cannot. If the correct answer requires a specific insight that the model rarely generates from scratch, even 100 independent samples may all miss it. But if you give the model a near-correct answer and ask it to identify and fix the specific flaw, the targeted feedback may guide it to the correct path. Refinement exploits the asymmetry between generation and verification in a more directed way than sampling.

The simplest form is self-refinement, where the model critiques its own output and revises it. A typical prompt sequence is:

1. Generation prompt: "Solve the following problem: {problem}"
2. Critique prompt: "Here is a proposed solution: {solution}. Identify any errors or weaknesses in the reasoning."
3. Refinement prompt: "Given this critique: {critique}, revise the solution to address the identified issues."

Steps 2 and 3 can repeat for several iterations, with each cycle consuming additional tokens and therefore compute. The model is acting as both solver and verifier, which works better than you might expect because generation and verification are distinct different cognitive tasks. A model may not generate a correct proof in one pass, but it may recognize that a specific step is invalid when asked directly. The critique role invokes a different part of the model's learned behavior than the generation role, and these two roles have different error rates on different parts of the problem.

The critical weakness of pure self-refinement is that the model's blind spots as a generator are correlated with its blind spots as a critic. If the model consistently makes a particular type of error, say, off-by-one errors in array indexing, it will tend to miss those same errors during critique as well. The model's knowledge gaps are persistent: they do not magically disappear when you ask the model to assume a different role. External feedback breaks this correlation by introducing a signal that is not derived from the same model that made the original error.

Another subtlety is that iterative self-refinement can sometimes make outputs worse rather than better. If the model's self-critique is inaccurate, it will "fix" aspects that were already correct, introducing new errors in the process. This happens when the critique task is harder than the original generation task, which can occur for complex, multi-step problems where the model cannot reliably reason about the correctness of intermediate steps. Research results on self-refinement by Madaan et al. (2023) document both the successes and the failures, and consistently find that structured external feedback outperforms pure self-critique.

A more reliable form of iterative refinement incorporates external signals that are independent of the model's own beliefs:

• Code execution feedback: Run the generated program against test cases and feed the error message back into the model. The error message is often more informative than anything the model could generate as a self-critique, because it identifies the exact line and nature of the failure rather than producing a general comment about code quality.
• Retrieval augmentation: After generating an initial answer, retrieve documents relevant to the specific claims made, then refine to incorporate the retrieved evidence. This is particularly useful for factual tasks where the model may confidently generate plausible-sounding but incorrect information.
• Formal verification feedback: For math proofs, feed the output into a proof assistant like Lean or Coq and return the specific error message. Even a partial proof that fails at a specific step gives the model much more precise information than a general-purpose self-critique.
• Human-in-the-loop feedback: Ask a human for a specific critique, then let the model incorporate it. This is expensive but often the highest-quality signal, and it is used in RLHF (Reinforcement Learning from Human Feedback) pipelines where the human preference signal is collected to train reward models.

The common thread across these feedback sources is that they are derived from a process that is structurally different from, and often more reliable than, the model itself. Running code against tests is a ground-truth check; retrieving documents accesses external knowledge the model may not have internalized; a human expert can catch errors that are invisible to the model. Each of these breaks the correlation between generator and critic that limits pure self-refinement.

A refinement loop can be formalized as a sequential process where each iteration updates the answer using the previous answer and new feedback. Starting from an initial answer $y^{(0)}$ generated from prompt $x$, each subsequent answer is produced by a refinement operator $f$:

$$y^{(t+1)} = f\!\left(x,\, y^{(t)},\, e^{(t)}\right)$$

where:

• $x$ is the original problem prompt, kept constant across all iterations
• $y^{(t)}$ is the answer at iteration $t$ (the current best solution)
• $e^{(t)}$ is the evaluation or feedback signal computed from $y^{(t)}$, such as an error message, a critique, or a reward score
• $f$ is the refinement function, implemented as a language model conditioned on all three inputs
• $y^{(T)}$ is the final answer after $T$ total refinement iterations

Under this formalization, the expected quality of $y^{(T)}$ after $T$ iterations is a function of the refinement operator's quality and the convergence rate of the process. A well-designed refinement operator converges rapidly; a poorly designed one may oscillate or diverge.

The refinement process has a natural analogy to gradient descent in weight space. Just as gradient descent iteratively updates model weights in the direction that reduces loss, iterative refinement updates the generated answer in the direction indicated by the feedback signal. The key difference is that refinement operates in the discrete space of language tokens rather than in the continuous space of model parameters, and the "gradient" is a natural language critique rather than a mathematically derived gradient vector.

This analogy helps clarify when refinement will converge. Gradient descent converges when the loss surface is well-behaved and the learning rate is appropriate. Refinement converges when the feedback signal accurately identifies the specific errors in the current answer and the model correctly implements the indicated changes. When the feedback is vague, the analogous "step" is poorly directed and convergence is slow. When the feedback is precise (a specific error message pointing to line 47 of a program), the analogous "step" is well-directed and convergence can be fast.

Iterative refinement is most effective when three conditions hold:

• Verifiability: There is a reliable way to evaluate the current answer. This can be automated (executing code, checking proofs) or model-based (using a well-trained critic), but the evaluation must be more reliable than the generation.
• Revisability: The answer space supports incremental improvement. Continuous outputs such as prose and code are easier to revise than discrete choices, because partial revisions make sense. You can fix one bug without rewriting the entire program; you cannot partially fix a multiple-choice answer selection.
• Budget surplus: The number of allowed tokens is large enough to sustain multiple critique-revise cycles. Each cycle approximately doubles the token cost relative to single-pass generation, so deep refinement chains are expensive.

Research results on self-refinement show mixed outcomes: for structured tasks with executable feedback, iterative refinement is highly effective and can convert a mediocre first-pass solution into a correct one within two or three cycles. For open-ended generation tasks without a clear correctness criterion, quality often plateaus after one or two iterations, and the model may introduce new errors while correcting old ones. The lesson is that refinement is not a universal solution but a powerful tool for specific problem types.