Reasoning Frontiers › Implementation: Simulating Test-Time Compute Scaling
Language AI HandbookAll chapters Reasoning Frontiers › Implementation: Simulating Test-Time Compute ScalingOne section · Sections in this chapter8 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The following code demonstrates key principles of test-time compute scaling through simulation. We build a simple problem-solving framework that illustrates best-of-N sampling, process reward estimation, and compute-quality tradeoffs. The simulation uses synthetic accuracy curves that capture the qualitative behavior documented in empirical scaling studies, without requiring access to actual reasoning models.
import random
import numpy as np
random.seed(42)
np.random.seed(42)We start by defining a simple problem class that simulates problems of varying difficulty and a mock "reasoner" whose accuracy depends on the number of reasoning steps it takes. The simulate_single_attempt function captures the core test-time scaling phenomenon: accuracy improves as a log-linear function of thinking tokens, with harder problems benefiting more from additional compute.
def simulate_single_attempt(difficulty, thinking_tokens):
"""
Simulate a single reasoning attempt.
difficulty: float in [0, 1], where 1 is hardest
thinking_tokens: int, number of tokens spent reasoning
Returns: (is_correct, tokens_used)
"""
# Base accuracy at zero thinking time
base_acc = max(0.05, 1.0 - difficulty)
# Accuracy improves with log of thinking tokens
# More thinking helps harder problems more
log_boost = 0.3 * difficulty * np.log1p(thinking_tokens / 100)
accuracy = min(0.98, base_acc + log_boost)
is_correct = random.random() < accuracy
return is_correct, thinking_tokensThe base_acc term sets a floor on accuracy determined by problem difficulty. An easy problem (difficulty = 0.3) starts at 70% accuracy even with minimal thinking. A hard problem (difficulty = 0.9) starts at only 10%. The log_boost term captures the test-time scaling improvement: each additional token of thinking helps, but with diminishing returns captured by the logarithm. Harder problems get a larger boost because there is more room for extended reasoning to help.
def best_of_n_sampling(difficulty, n, tokens_per_attempt):
"""
Simulate best-of-N sampling: generate N independent attempts
and select via majority vote (oracle verifier).
Returns: (is_any_correct, total_tokens_used)
"""
results = [
simulate_single_attempt(difficulty, tokens_per_attempt)
for _ in range(n)
]
any_correct = any(r[0] for r in results)
total_tokens = sum(r[1] for r in results)
return any_correct, total_tokensdef sequential_reasoning(difficulty, total_tokens):
"""
Simulate sequential reasoning: use all tokens in one long chain.
The single attempt gets the full compute budget.
Returns: (is_correct, tokens_used)
"""
return simulate_single_attempt(difficulty, total_tokens)Now let's measure how accuracy scales with compute budget for problems of different difficulties, comparing best-of-N versus sequential reasoning.
difficulties = [0.3, 0.6, 0.9] # Easy, medium, hard
compute_budgets = [100, 300, 1000, 3000, 10000, 30000]
n_trials = 500
from collections import defaultdict
results = defaultdict(lambda: defaultdict(dict))
for difficulty in difficulties:
for budget in compute_budgets:
# Best-of-N: split budget into N attempts of equal size
# Try several values of N and pick best
best_n_acc = 0.0
for n in [1, 2, 4, 8, 16, 32]:
tokens_per = max(50, budget // n)
actual_n = max(1, budget // tokens_per)
correct = sum(
best_of_n_sampling(difficulty, actual_n, tokens_per)[0]
for _ in range(n_trials)
)
acc = correct / n_trials
best_n_acc = max(best_n_acc, acc)
results[difficulty][budget]["best_of_n"] = best_n_acc
# Sequential: all tokens in one attempt
correct = sum(
sequential_reasoning(difficulty, budget)[0] for _ in range(n_trials)
)
results[difficulty][budget]["sequential"] = correct / n_trials
Easy (difficulty=0.3):
Budget | Sequential | Best-of-N
-----------------------------------
100 | 75.4% | 93.8%
300 | 83.8% | 100.0%
1,000 | 91.8% | 100.0%
3,000 | 97.0% | 100.0%
10,000 | 98.0% | 100.0%
30,000 | 98.2% | 100.0%
Medium (difficulty=0.6):
Budget | Sequential | Best-of-N
-----------------------------------
100 | 51.2% | 73.2%
300 | 63.2% | 98.6%
1,000 | 82.0% | 100.0%
3,000 | 98.4% | 100.0%
10,000 | 98.8% | 100.0%
30,000 | 97.2% | 100.0%
Hard (difficulty=0.9):
Budget | Sequential | Best-of-N
-----------------------------------
100 | 30.0% | 44.2%
300 | 46.6% | 77.2%
1,000 | 74.2% | 99.0%
3,000 | 97.6% | 100.0%
10,000 | 97.6% | 100.0%
30,000 | 98.4% | 100.0%Both strategies improve with compute. For hard problems, best-of-N tends to outperform sequential reasoning at high compute budgets because parallel exploration finds correct paths that sequential reasoning might miss when a single long attempt goes down an unproductive direction.
Now let's visualize how the optimal N in best-of-N sampling changes as a function of problem difficulty and compute budget.
def optimal_n_experiment(difficulty, total_budget, n_values, n_trials=300):
"""Find the best N for a given difficulty and compute budget."""
best_n = 1
best_acc = 0.0
for n in n_values:
tokens_per = max(50, total_budget // n)
correct = sum(
best_of_n_sampling(difficulty, n, tokens_per)[0]
for _ in range(n_trials)
)
acc = correct / n_trials
if acc > best_acc:
best_acc = acc
best_n = n
return best_n, best_acc
n_values_to_test = [1, 2, 4, 8, 16, 32, 64]
fixed_budget = 10000
optimal_n_results = {}
for d in difficulties:
optimal_n_results[d] = {}
for n in n_values_to_test:
tokens_per = max(50, fixed_budget // n)
correct = sum(
best_of_n_sampling(d, n, tokens_per)[0] for _ in range(400)
)
optimal_n_results[d][n] = correct / 400
This tradeoff illustrates a core insight from the test-time compute scaling literature: harder problems benefit from higher N because the probability that at least one attempt succeeds increases with N, but each attempt needs enough tokens to have a non-trivial success probability. There is an optimal N for each difficulty level and compute budget, and finding that optimum is a non-trivial problem in practice.
Now let's simulate a process reward model (PRM) scenario, showing how step-level verification enables more efficient use of compute compared to outcome-only evaluation.
def simulate_prm_guided_search(difficulty, max_tokens, n_steps=5, n_trials=300):
"""
Simulate PRM-guided search vs. outcome-only evaluation.
PRM guidance: prune bad paths early, concentrating compute on promising ones.
Outcome-only: use all tokens and evaluate at the end.
Returns: (prm_accuracy, outcome_accuracy, prm_avg_tokens, outcome_avg_tokens)
"""
tokens_per_step = max_tokens // n_steps
prm_correct = 0
outcome_correct = 0
prm_tokens_used = []
outcome_tokens_used = []
# PRM error detection rate (probability of catching a bad step)
prm_detection_rate = 0.7
for _ in range(n_trials):
# Outcome-only: just run the full reasoning chain
is_correct, tokens = simulate_single_attempt(difficulty, max_tokens)
outcome_correct += int(is_correct)
outcome_tokens_used.append(tokens)
# PRM-guided: evaluate each step and prune if step seems bad
cumulative_tokens = 0
step_correct = True
for step in range(n_steps):
cumulative_tokens += tokens_per_step
# Probability that this step has an error
step_error_prob = difficulty * 0.4
step_has_error = random.random() < step_error_prob
if step_has_error and random.random() < prm_detection_rate:
# PRM correctly detected the error: prune this path
step_correct = False
break
if step_correct:
# Completed all steps without pruning: evaluate final answer
# Better outcome probability due to error-free steps
final_acc = min(0.98, (1.0 - difficulty) + 0.4 * (1.0 - difficulty))
is_correct = random.random() < final_acc
prm_correct += int(is_correct)
# If pruned, we got it wrong (would retry in practice)
prm_tokens_used.append(cumulative_tokens)
prm_acc = prm_correct / n_trials
outcome_acc = outcome_correct / n_trials
avg_prm_tokens = np.mean(prm_tokens_used)
avg_outcome_tokens = np.mean(outcome_tokens_used)
return prm_acc, outcome_acc, avg_prm_tokens, avg_outcome_tokens
# Run experiment across difficulties
prm_experiment = {}
for d in difficulties:
results_prm = simulate_prm_guided_search(d, max_tokens=2000, n_trials=500)
prm_experiment[d] = {
"prm_acc": results_prm[0],
"outcome_acc": results_prm[1],
"prm_tokens": results_prm[2],
"outcome_tokens": results_prm[3],
}Process Reward Model vs. Outcome-Only Evaluation ============================================================ Difficulty | Outcome Acc | PRM Acc | Avg PRM Tokens ------------------------------------------------------------ Easy (0.3) | 97.6% | 61.2% | 1679 Medium (0.6) | 94.0% | 22.8% | 1462 Hard (0.9) | 91.4% | 2.6% | 1153
The PRM-guided approach uses fewer tokens on average by pruning poor reasoning paths early, illustrating the efficiency gains that process reward models provide.
