Inference Scaling › Inference-Optimal Model Selection

Given expected inference demand, we can determine the optimal model size directly. This requires understanding how loss scales with both model size and training data, and then reformulating the optimization problem to account for total lifecycle costs rather than just training costs.

Building on the Chinchilla loss function from earlier chapters:

$$L(N, D) = \frac{A}{N^\alpha} + \frac{B}{D^\beta} + E$$

where:

• $L(N, D)$: the model's loss as a function of size and training data
• $N$: the number of model parameters
• $D$: the number of training tokens
• $A, B$: empirically fitted scaling coefficients
• $\alpha, \beta$: empirically fitted exponents (typically around 0.34 and 0.28)
• $E$: the irreducible error floor representing the entropy of natural language

This functional form captures two distinct pathways to reducing loss: adding more parameters (the first term) or training on more data (the second term). The irreducible error floor $E$ is the basic uncertainty in language. Even a perfect model cannot predict with certainty what word comes next in human text. Understanding this structure is needed for reasoning about the trade-offs between model size and training duration.

We can reformulate the optimization problem for inference-dominant scenarios. Rather than minimizing loss for a fixed training budget (the Chinchilla approach), we want to minimize loss for a fixed total budget that includes both training and expected inference. This reformulation shifts the optimization target from training efficiency to deployment efficiency.

The total compute over the model's lifecycle is:

$$C_{total} = C_{train} + C_{inference} = 6ND + 2N \cdot T_{inference}$$

where:

• $C_{total}$: total compute over the model's entire lifecycle
• $C_{train} = 6ND$: training compute (approximately 6 FLOPs per parameter per token)
• $C_{inference} = 2N \cdot T_{inference}$: total inference compute (2 FLOPs per parameter per token generated)
• $T_{inference}$: expected total tokens generated during the model's deployment

This formulation captures the key insight: training cost is paid once, but inference cost accumulates with every token generated. The training term depends on both $N$ and $D$ (model size times data), while the inference term depends on $N$ and $T_{inference}$ (model size times usage). This asymmetric structure drives the preference for smaller models in high-inference scenarios.

To minimize loss subject to a total compute budget, we take derivatives and solve. The resulting optimal allocation shifts toward smaller models as expected inference volume increases:

$$N^* \propto C_{total}^{a} \cdot T_{inference}^{-b}$$

where:

• $N^*$: the optimal model size (number of parameters) for the given deployment scenario
• $C_{total}$: the total compute budget available across training and inference
• $T_{inference}$: expected total tokens to be generated during deployment
• $a, b$: positive exponents derived from the scaling law parameters $\alpha$ and $\beta$. The exact values depend on the specific scaling law coefficients, but typical values yield $a \approx 0.5$ and $b \approx 0.3$

The negative exponent on $T_{inference}$ is the key result: it shows mathematically that optimal model size decreases as expected inference volume increases. This relationship emerges directly from the calculus of constrained optimization. When inference tokens grow, the cost of each parameter (which must be activated for every token) grows proportionally, pushing the optimum toward fewer parameters.

The intuition is clear: as inference demand grows, the optimal model size shrinks because each parameter incurs a cost at every inference, while training cost is amortized. A parameter that costs billions of FLOPs to train well might seem expensive, but if that parameter is then used trillions of times at inference, the training cost becomes negligible compared to the cumulative inference burden.

For deployment planning, a simplified heuristic helps guide model selection. If you expect to generate $T$ inference tokens over the model's lifetime, compare the inference compute to training compute:

$$\text{Inference factor} = \frac{2N \cdot T}{6ND} = \frac{T}{3D}$$

where:

• $T$: total inference tokens expected over the model's deployment lifetime
• $D$: number of tokens used for training
• $N$: model parameters (which cancel out in the simplification)

Notice that $N$ cancels out. The inference factor depends only on the ratio of inference tokens to training tokens. This cancellation is mathematically elegant and practically useful because you can evaluate whether inference will dominate without even knowing the model size. When this factor exceeds 1, inference compute dominates. The larger this factor, the more aggressively you should favor smaller, over-trained models.

The inference factor gives a simple decision rule. Values below 1 indicate training-dominated scenarios where Chinchilla-optimal sizing remains appropriate. Values above 1 signal that inference costs will outweigh training costs, justifying investment in smaller, more heavily trained models. Values above 10 suggest aggressive over-training is warranted. Values above 100 indicate that inference efficiency should be the primary design consideration.