Test-Time Compute › Compute-Optimal Inference

You have a fixed inference budget of $B$ total tokens, or equivalently, a fixed number of FLOPS beyond the base model cost. How should you allocate them? Should you generate $N$ independent samples of fixed length, or should you generate fewer samples but allow each one to use a longer chain of thought? Should you run $T$ refinement iterations, or use the same tokens for a single longer generation?

These questions define the compute-optimal inference problem: given a budget $B$ and a task, find the allocation strategy that maximizes expected output quality. This is a practical question. In production systems, the inference budget is directly proportional to cost and inversely proportional to throughput. Getting the allocation right can mean the difference between a system that is economically viable and one that is not.

The problem is also more subtle than it appears. The optimal allocation is not a fixed property of the model; it depends on the task type, the model's base accuracy, the quality of the selection mechanism, and the relationship between chain length and per-sample accuracy. Understanding how these factors interact is the key to designing efficient inference systems.

The fundamental tradeoff is between breadth (more parallel samples) and depth (longer reasoning chains per sample). Let $B$ be the total token budget. If you generate $N$ samples each of length $L$, then the budget constraint is:

where:

• $B$ is the total token budget (fixed)
• $N$ is the number of independent samples
• $L$ is the length of each sample in tokens

This constraint defines a tradeoff curve: holding $B$ fixed, increasing $N$ forces $L$ to decrease, and vice versa. The extreme points are:

where $L_{\min}$ is the minimum useful answer length for the task. The optimal point along this curve depends on the task. For verifiable tasks with low per-sample accuracy, breadth tends to win at small budgets because you need many attempts to land on a correct solution. For tasks where structured reasoning chains substantially improve per-sample accuracy, depth tends to win because the conditional accuracy given a well-formed chain is much higher.

To see why, consider the expected accuracy under each extreme. For the all-breadth strategy with $N = B / L_{\min}$ samples of fixed minimal length, each sample has accuracy $p_{\min}$, the accuracy achievable in $L_{\min}$ tokens:

For the all-depth strategy with a single sample of length $B$, the accuracy is $p_{\max}$, the accuracy achievable with the full budget in one chain:

Breadth wins when $p_{\min}$ is low but $N = B / L_{\min}$ is large enough that the combined probability of at least one correct sample exceeds $p_{\max}$. Depth wins when the full-chain accuracy $p_{\max}$ is much higher than $p_{\min}$, meaning the chain-of-thought reasoning provides substantial value that cannot be replicated by sampling more short chains.

The two curves in this plot represent qualitatively different problem types, and they point toward opposite allocation strategies. The breadth-wins curve rises monotonically because the model's per-sample accuracy does not depend on chain length: it is the kind of problem where the answer is either in the model's knowledge or it is not, and more reasoning time does not help much. Short, independent samples are the right tool. The depth-wins curve shows that allocating all compute to a single long chain yields the highest accuracy, because the per-sample accuracy collapses when chains are too short to complete the reasoning.

In practice, most real tasks fall somewhere between these extremes. A math problem that requires five reasoning steps benefits from some minimum chain length (you need at least enough tokens to complete the derivation), but additional breadth helps once that minimum is satisfied. Identifying where a specific task sits on this spectrum requires empirical measurement on a held-out validation set.

A key empirical question is whether inference compute exhibits scaling laws analogous to training compute laws. For a fixed model and task distribution, the relationship between the number of generated samples $N$ and the probability of a correct answer follows an approximate power law. An empirical fit often has the form:

where:

• $N$ is the number of samples (or equivalent token budget)
• $a > 0$ is a scale coefficient reflecting the marginal value of each additional sample at the task
• $b \in (0, 1)$ is the scaling exponent, typically observed in the range 0.1 to 0.3, reflecting sublinear returns
• $c \ge 0$ is the baseline accuracy floor at $N = 1$ (single-pass greedy decoding)

The exponent $b$ being in the range $(0, 1)$ means doubling the compute budget gives a smaller relative gain each time. Going from 1 to 2 samples improves accuracy more than going from 64 to 128 samples. This matches the intuition from the pass@k formula: early samples cover the most likely correct answers; later samples fill in increasingly unlikely edge cases.

The existence of power-law scaling at inference time is significant because it means inference compute obeys the same mathematical regularity as training compute. Just as the Chinchilla scaling laws (covered in the Scaling Laws chapter) predict model quality from training budget, inference scaling laws can in principle predict answer quality from inference budget for a given task and model. This makes it possible to reason systematically about the tradeoff between training and inference compute rather than treating each as a separate, independent choice.

Work by Snell et al. (2024) shows that a smaller model with compute-scaled inference can match or exceed a larger model running greedy decoding on hard reasoning tasks. A model with 3.8B parameters using 256 samples with PRM reranking can match a 70B model with single-pass decoding on MATH benchmarks. This "compute equivalence" result suggests that inference compute and parameter count are partially substitutable, at least for tasks with verifiable structure. The practical implication is that teams with limited hardware can potentially close quality gaps by spending more on inference rather than training or serving larger models.

The most efficient inference systems route problems to different compute budgets based on estimated difficulty. Instead of applying the same sampling strategy to every query, adaptive allocation reserves expensive multi-sample strategies for the queries that need them, and handles easy queries with cheap single-pass decoding.

Several mechanisms implement adaptive allocation:

Difficulty estimation. Use a fast heuristic, such as few-shot scoring, a lightweight classifier, or the model's own uncertainty estimate, to predict whether a question is easy, medium, or hard before full generation. Allocate budget proportionally. Even an imperfect difficulty classifier substantially reduces average cost if it correctly identifies the easy queries that dominate most real-world distributions.

Early exit. If the model reaches high-confidence agreement across a small $N$ (say, 5 out of 5 candidates agree), stop early without using the full budget. The stopping criterion can be any measure of candidate consensus: exact string match for mathematical answers, semantic similarity for prose, or a threshold on the reward model's score distribution.

Progressive refinement. Start with $N = 4$ samples. If the selector is uncertain (for example, a close vote between two candidate answers), increase to $N = 16$, then $N = 64$. This amortizes the cost of easy questions while preserving quality on hard ones. The model effectively makes a cascade of decisions: "Is this problem hard enough to warrant more compute?"

OpenAI's o1 and o3 family of models applies an implicit version of this strategy: they reason internally using a chain-of-thought scratchpad before producing an answer, spending more tokens on harder problems. The o3 model's exceptional performance on ARC-AGI benchmarks demonstrated that inference compute scaling can overcome task difficulties that large parameter counts alone cannot address. The model spends orders of magnitude more inference compute on the hardest problems than on routine queries, effectively self-routing its own budget.

The most interesting practical observation from adaptive systems is that problem difficulty is approximately predictable. A fast classifier trained on the question text alone can distinguish easy from hard questions with useful accuracy. This means the adaptive routing overhead is small compared to the budget savings on easy questions. In production systems, routing the easiest 70% of queries to single-pass decoding and reserving aggressive multi-sample strategies for the hard 30% can cut total inference cost in half with minimal quality loss on the overall distribution.

Adaptive allocation also changes the economics of test-time compute. A uniform multi-sample strategy applied to all queries is expensive and may not be justifiable for most applications. An adaptive strategy that targets the expensive resources at hard queries can achieve comparable quality at a fraction of the cost. This is the design philosophy behind modern reasoning models, and it represents one of the most important practical advances in efficient inference.