Reasoning Verification › Self-Correction

A conceptually appealing idea is to have a model verify and correct its own outputs: generate an initial answer, identify errors in it, and revise accordingly. This is called self-correction or self-refinement. It requires no external verifier, just the model's own ability to evaluate its reasoning.

The appeal of self-correction is practical. External verifiers require separate training, separate inference, and maintenance as the generator model improves. If a generator model could reliably detect and fix its own errors, verification would become much cheaper. The question is whether this capability exists, and under what conditions.

Effective self-correction has several prerequisites:

• The model must be able to detect errors in its own output that it could not detect at generation time. This requires either a different reasoning process during review than during generation, or information that was not available during the initial generation.
• The model must be able to generate corrections that fix those errors, not just paraphrase the flawed reasoning in different words.
• The corrections must not introduce new errors while fixing existing ones.

These requirements are stringent. If the model was confident and wrong during initial generation, it may also be confident and wrong during self-evaluation. Research has shown that naive self-correction, where a model is simply asked to review and improve its answer, often fails to improve accuracy and can even decrease it. The model tends to second-guess correct answers, introduce unnecessary complexity, or produce superficial changes that do not address underlying errors.

The fundamental problem is that the model's internal state during generation already reflected the best representation it could form given its weights and the context. Asking it to "review" that output does not provide new information unless the prompt structure changes the effective inference procedure. Simply appending "please check your work" to the prompt and sampling again draws from a similar distribution to the original output.

Self-correction is most effective in several specific conditions.

The model has access to external feedback. When a code interpreter can run the model's code and return an error message, the model can incorporate that specific external signal. This is not pure self-correction but verification-grounded correction. The external executor provides information that was not available during generation, which the model can use to diagnose and fix the error. Research on code generation consistently finds that execution-based self-correction substantially improves success rates on programming tasks.

The error type is systematic and detectable. If a model reliably can detect format violations, constraint violations, or factual inconsistencies against its own earlier statements, it can correct them. A model that generates a list of five items when the prompt requested exactly three can detect and fix this count error. It is harder to self-correct subtle logical errors that require deep domain reasoning to identify.

The initial generation used a different strategy than correction. If the model generates a concise answer first and then asks itself to work through the problem step by step, the second pass may catch errors the first pass made by taking shortcuts. The change in reasoning strategy (rather than another review of the same strategy) is what enables improvement. Some papers call this "draft-then-verify" and show it can improve accuracy relative to generating the chain of thought directly.

Confidence is calibrated. A model that knows when it is uncertain can focus correction effort on uncertain steps. If confidence is uncalibrated (the model is equally confident when right and wrong), self-verification provides no useful signal. Well-calibrated models can use their own uncertainty estimates to identify which steps are worth re-examining.

The problem has verifiable intermediate structure. Word problems with numerical answers, code problems with test cases, and logic puzzles with checkable constraints all provide internal checkability. After generating a candidate solution, the model can re-read the problem statement and check whether each claimed step follows from the stated premises.

Iterative self-refinement applies self-correction in a loop: the model generates an output, evaluates it, generates feedback on what is wrong, and produces an improved version. This loop continues for a fixed number of iterations or until the model assesses its output as satisfactory.

The procedure at each iteration $t$ is:

1. Generate output $y^{(t)}$ given the problem $x$ and prior output $y^{(t-1)}$
2. Generate feedback $f^{(t)}$ by prompting the model to evaluate $y^{(t)}$
3. Generate revised output $y^{(t+1)}$ given $x$, $y^{(t)}$, and $f^{(t)}$

The key question is whether $y^{(t+1)}$ is systematically better than $y^{(t)}$. For tasks with clear objective criteria (code that runs, outputs that satisfy explicit constraints), iterative refinement tends to improve output quality. For tasks with subjective criteria or where model errors are subtle, the improvement is less consistent, and gains are often small relative to the additional compute cost.

One important failure mode in iterative self-refinement is "overcorrection." The model may fix an actual error in round $t$ but then correct a correct element in round $t+1$, oscillating rather than converging to a correct answer. This is especially common when the model is asked to evaluate its own confidence in an uncalibrated way. Adding a stopping criterion based on agreement between consecutive outputs (if $y^{(t)}$ and $y^{(t+1)}$ differ only superficially, stop) can reduce wasted iterations without improving quality.

The practical difference between self-correction and external verification is that self-correction uses the same model for generation and evaluation, while external verification uses a separate model or system trained specifically to evaluate reasoning quality.

External verifiers have a clear advantage: they are trained for evaluation, not generation, so their evaluation may be more reliable. A model fine-tuned specifically on step correctness labels is better at detecting step errors than the same base model prompted to self-evaluate. The PRM has seen thousands of examples of incorrect steps alongside correct ones during training; the base model asked to self-evaluate has no such specialized training.

The trade-off is cost and generality. External verifiers require separate training or separate model calls. Self-correction requires only additional inference from the same model. For many practical applications, a combination works well: use the model itself for initial self-filtering (removing outputs that fail obvious checks), then use an external PRM for final selection among the remaining candidates.