Test-Time Compute › Code Implementation: Iterative Refinement Simulation
Language AI HandbookAll chapters Test-Time Compute › Code Implementation: Iterative Refinement SimulationOne section · Sections in this chapter7 / 10 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Now let's simulate iterative refinement and measure how quality evolves with each iteration. This demonstrates the key dynamics of the refinement process: the initial gain from the first revision, the diminishing returns in later iterations, and the convergence properties of different model configurations.
Refinement Model
class RefinementModel:
"""
Simulate a model that improves answers iteratively.
At each step, the model has a base probability of being correct,
plus a refinement gain that decays geometrically.
"""
def __init__(
self,
base_accuracy: float = 0.3,
refinement_gain: float = 0.15,
decay_factor: float = 0.6,
seed: int = 42,
):
self.base_accuracy = base_accuracy
self.refinement_gain = refinement_gain
self.decay_factor = decay_factor
self.rng = np.random.default_rng(seed)
def effective_accuracy(self, iteration: int) -> float:
"""
Compute the accumulated accuracy after a given number of refinement steps.
The accuracy at iteration t combines the base accuracy with the sum of
geometric refinement gains up to that point:
P(correct at t) = base + gain * (1 - decay^t) / (1 - decay)
This is a geometric series sum, capped at 1.0.
Args:
iteration: The refinement iteration index (0 = initial generation).
Returns:
The expected accuracy at this iteration.
"""
if iteration == 0:
return self.base_accuracy
geometric_sum = (
self.refinement_gain
* (1 - self.decay_factor**iteration)
/ (1 - self.decay_factor)
)
return min(1.0, self.base_accuracy + geometric_sum)
def refine(self, iteration: int, n_questions: int = 5000) -> float:
"""Simulate accuracy at the given refinement iteration."""
p = self.effective_accuracy(iteration)
correct = self.rng.binomial(1, p, n_questions)
return float(correct.mean())Refinement accuracy over iterations (base=30%)
Iteration Theoretical P Simulated Acc
---------------------------------------------
0 0.300 0.294
1 0.420 0.421
2 0.486 0.489
3 0.522 0.512
4 0.542 0.546
5 0.553 0.550
6 0.559 0.566The simulated accuracy follows the theoretical curve closely. The refinement gain decreases geometrically: the first iteration yields the largest improvement, and subsequent iterations yield diminishing returns. This geometric decay is not arbitrary; it reflects the intuition that early revisions fix the most obvious errors, while later revisions address progressively subtler issues that become harder to identify and correct.
The decay_factor parameter controls how quickly returns diminish. A decay factor near
0 means each iteration provides essentially the same gain as the first (unrealistic in
practice). A decay factor near 1 means each iteration provides nearly the same gain
as the one before, implying slow convergence. The observed range in practice, roughly
0.4 to 0.7, means that the second iteration yields about half the gain of the first,
and the third iteration yields about a quarter.
Comparing Multiple Sampling vs. Refinement
A natural question is whether a fixed token budget is better spent on multiple independent samples or on iterative refinement. This is the empirical version of the breadth-depth tradeoff question: multiple sampling increases breadth, while iterative refinement increases effective depth. Let's simulate both strategies under a shared compute budget and observe where each excels.
def simulate_budget_comparison(
n_questions: int = 3000,
budget_tokens: list = None,
tokens_per_sample: int = 200,
tokens_per_refinement: int = 150,
base_acc: float = 0.28,
seed: int = 7,
) -> dict:
"""
Compare multiple sampling vs. iterative refinement under token budgets.
Multiple sampling: budget = N * tokens_per_sample, N = budget / tokens_per_sample.
Refinement: budget = iterations * tokens_per_refine, T = budget / tokens_per_refine.
"""
if budget_tokens is None:
budget_tokens = [200, 400, 800, 1600, 3200, 6400]
# Multiple sampling model
sampling_model = SimulatedModel(true_accuracy=base_acc, seed=seed)
# Refinement model
refine_model = RefinementModel(
base_accuracy=base_acc,
refinement_gain=0.10,
decay_factor=0.5,
seed=seed,
)
sampling_accs = []
refinement_accs = []
for budget in budget_tokens:
n_samples = max(1, budget // tokens_per_sample)
acc_sampling = evaluate_majority_vote(
sampling_model, n_questions, n_samples, seed=seed
)
sampling_accs.append(acc_sampling)
n_iterations = max(0, budget // tokens_per_refinement - 1)
acc_refine = refine_model.refine(n_iterations, n_questions)
refinement_accs.append(acc_refine)
return {
"budget_tokens": budget_tokens,
"sampling": sampling_accs,
"refinement": refinement_accs,
}
budget_comparison = simulate_budget_comparison()Accuracy vs. Token Budget: Sampling vs. Refinement
Budget Sampling Refinement Difference
------------------------------------------------
200 0.301 0.284 + 0.017
400 0.277 0.372 -0.096
800 0.294 0.485 -0.190
1600 0.318 0.484 -0.167
3200 0.345 0.489 -0.143
6400 0.385 0.477 -0.092The relative advantage of each strategy depends on the budget magnitude and the task's properties. Multiple sampling tends to dominate at moderate budgets when per-sample accuracy is low; refinement dominates when the model's refinement operator is highly effective and the feedback signal is reliable. The crossover point shifts with model quality: stronger models that generate higher-quality first-pass answers get less benefit from parallel sampling but more benefit from targeted refinement, because their first-pass answers are close enough to correct that a focused critique can bridge the gap.
Key Parameters
The core parameters governing test-time compute strategies are:
- N (number of samples): The number of independent candidates generated. Higher N improves coverage but increases latency proportionally.
- k (selection budget): The number of samples passed to the selector. Often equal to N, but can be smaller if you want faster selection.
- T (refinement iterations): The number of critique-revise cycles. Each iteration consumes tokens proportional to the answer length plus the critique.
- Temperature: Controls diversity in parallel sampling. Higher temperature gives more diverse candidates; lower temperature reduces variance but also coverage.
- Selector type: Majority vote, reward model, PRM, or verifier. Each has a different quality-cost tradeoff.