Inference Scaling › Why Chinchilla is Training-Optimal, Not Deployment-Optimal

Recall from the Chinchilla scaling laws chapter that compute-optimal training allocates compute roughly equally between model size and data, following the relationship $D \approx 20N$ where $D$ is tokens trained and $N$ is parameters. This minimizes training FLOPs to reach a target loss. This ratio of roughly 20 tokens per parameter comes from empirical study and gives a clear recipe for efficient training.

But the Chinchilla framework optimizes for a specific objective: minimizing the compute required to reach the lowest possible loss during training. It treats training as the end goal rather than as a means to an end. When we recognize that the true objective is efficient deployment, not efficient training, the optimal strategy shifts sharply.

Consider two models trained to the same final loss:

Model A (Chinchilla-optimal):

• 70B parameters
• 1.4T tokens
• Training compute: $C_A = 6ND = 6 \times 70 \times 10^9 \times 1.4 \times 10^{12} \approx 5.9 \times 10^{23}$ FLOPs

The $6ND$ approximation comes from the fact that each training token requires a forward pass ($2N$ operations) plus a backward pass (approximately $4N$ operations for computing gradients), totaling $6N$ operations per token across $D$ tokens. This 6-to-1 ratio between training and inference compute per token explains why training is so much more expensive than inference on a per-token basis, but also why inference costs accumulate so quickly when serving many users.

where:

• $N$: number of model parameters
• $D$: number of training tokens
• $6ND$: total FLOPs, accounting for forward pass (2 operations per parameter per token) plus backward pass (roughly 4 operations per parameter per token)

Model B (Over-trained):

• 7B parameters
• 4T tokens (requires longer training to match Model A's loss)
• Training compute: $C_B = 6 \times 7 \times 10^9 \times 4 \times 10^{12} = 1.68 \times 10^{23}$ FLOPs

Model B uses less training compute to reach the same loss. How can this be? The key is that Chinchilla optimality assumes you're pushing to the frontier of what's achievable at a given compute level. If you're willing to accept a specific target loss rather than minimizing loss for fixed compute, smaller models trained longer can sometimes be more efficient even for training.

This counterintuitive result arises because Chinchilla optimality answers a different question than the one we're asking. Chinchilla asks: "Given a fixed compute budget, how do I get the lowest possible loss?" The deployment question asks: "Given a target quality level, how do I minimize total cost over the model's lifetime?" These are fundamentally different optimization problems with different optimal solutions.

The real advantage of smaller models emerges at inference time. Model A requires $2 \times 70 \times 10^9 = 1.4 \times 10^{11}$ FLOPs per output token, while Model B requires $2 \times 7 \times 10^9 = 1.4 \times 10^{10}$ FLOPs per token. This means 10× fewer inference FLOPs for equivalent quality.

This 10× reduction in per-token cost compounds with every inference request. If both models serve a billion requests, Model B saves approximately $1.26 \times 10^{20}$ FLOPs per token generated. This compute savings translates directly to reduced electricity bills, fewer GPUs required, and lower operational costs.

For high-volume deployments, this 10× inference efficiency can translate to large cost savings that dwarf any training cost difference.