Inference Scaling › Over-Training for Deployment Efficiency

"Over-training" refers to training a model on more tokens than Chinchilla optimality would suggest. The term is slightly misleading, as it's only "over" training relative to compute-optimal. It's exactly the right amount of training for inference-optimal deployment. Think of it not as excessive training but as investing additional training compute to reduce inference costs later.

The LLaMA model family explicitly adopted this philosophy. The LLaMA paper states: "We only consider the performance of the resulting model, not the training compute budget." They trained 7B, 13B, 33B, and 65B parameter models on up to 1.4 trillion tokens. This is far more than Chinchilla ratios would suggest for the smaller models.

This statement is a philosophical break from the Chinchilla paradigm. Rather than asking how to minimize training compute, Meta's researchers asked how to maximize the value delivered per inference FLOP. This reframing led to models that were "inefficient" to train but were much more efficient to deploy.

For the 7B model, Chinchilla optimality would suggest around 140B tokens. Training on 1.4T tokens is 10× "over-training." This extra training compute is an investment that pays dividends at inference time through reduced per-token cost. Every additional training token helps the smaller model close the capability gap with larger alternatives, which makes it suitable for tasks that would otherwise require a more expensive model.

We can quantify over-training with a simple ratio that captures how far a model departs from Chinchilla-optimal training:

$$\text{Over-training ratio} = \frac{D_{actual}}{D_{optimal}} = \frac{D_{actual}}{20N}$$

where:

• $D_{actual}$: the number of tokens used to train the model
• $D_{optimal}$: the Chinchilla-optimal number of tokens for a model of size $N$
• $N$: the number of model parameters
• $20N$: the Chinchilla approximation that optimal training uses roughly 20 tokens per parameter

This ratio gives a simple metric for characterizing how inference-optimized a model's training regime is. A ratio of 1 means the model was trained according to Chinchilla-optimal principles. Ratios greater than 1 indicate over-training for inference efficiency, with higher values representing more aggressive investment in training compute to reduce inference costs.

The table below shows typical over-training ratios for inference-optimized models:

Modern inference-optimized models routinely use over-training ratios of 10× to 50× or more. The trend toward higher ratios shows growing recognition that inference costs dominate for production systems. Mistral's 57× over-training ratio on their 7B model shows how far this philosophy has been pushed. They invested roughly 57 times more training compute than Chinchilla would recommend. They bet that the resulting inference efficiency would more than compensate.

Over-training makes economic sense when total inference compute exceeds the extra training compute required. Let's formalize this break-even analysis.

Suppose you have two options:

• Option 1: Train a large model $N_1$ on $D_1$ tokens (Chinchilla-optimal)
• Option 2: Train a smaller model $N_2 < N_1$ on $D_2 > D_1$ tokens to match Option 1's loss

The additional training cost for Option 2 is:

$$\Delta C_{train} = 6N_2D_2 - 6N_1D_1$$

where:

• $\Delta C_{train}$: the difference in training compute between the two options
• $N_1, N_2$: parameter counts for Option 1 (larger) and Option 2 (smaller) respectively
• $D_1, D_2$: training tokens for Option 1 and Option 2 respectively

This difference can be positive or negative depending on the specific model sizes and training durations. If the smaller model requires enough additional training to offset the reduced parameter count, $\Delta C_{train}$ will be positive, representing an upfront investment that must be recouped through inference savings.

The inference savings per token generated are:

$$\Delta C_{inference} = 2(N_1 - N_2)$$

Here, $\Delta C_{inference}$ is the FLOPs saved per output token by using the smaller model (since each forward pass requires $2N$ FLOPs per token).

This quantity is always positive when $N_1 > N_2$, meaning smaller models always save inference compute regardless of how they were trained. The magnitude of these savings grows linearly with the difference in model sizes. A 63B parameter reduction saves 63 billion fewer multiply-add operations per token.

Over-training pays off when total inference tokens $T_{inference}$ satisfies:

$$T_{inference} \times \Delta C_{inference} > \Delta C_{train}$$ $$T_{inference} > \frac{6N_2D_2 - 6N_1D_1}{2(N_1 - N_2)}$$

This break-even analysis determines whether the inference savings justify the training investment. The inequality has a clear interpretation: the left side is cumulative inference savings over the model's deployment, while the right side is the upfront training cost differential. When expected inference volume exceeds the break-even point, the smaller over-trained model becomes the economically rational choice.

Let's work through a concrete comparison between a 70B Chinchilla-optimal model and a 7B over-trained model, both achieving similar loss. This example illustrates the practical magnitude of the cost differences and helps build intuition for deployment decisions.

Option 1: 70B model, 1.4T tokens

• Training: $6 \times 70B \times 1.4T = 5.9 \times 10^{23}$ FLOPs
• Inference: $2 \times 70B = 1.4 \times 10^{11}$ FLOPs/token

Option 2: 7B model, 4T tokens (assume this matches 70B model's loss)

• Training: $6 \times 7B \times 4T = 1.68 \times 10^{23}$ FLOPs
• Inference: $2 \times 7B = 1.4 \times 10^{10}$ FLOPs/token

In this case, Option 2 costs less to train. This can happen when targeting a specific loss level rather than frontier performance. The inference savings are:

$$\Delta C = 1.4 \times 10^{11} - 1.4 \times 10^{10} = 1.26 \times 10^{11} \text{ FLOPs/token}$$

Every inference token saves $1.26 \times 10^{11}$ FLOPs, and there's no training penalty. This is a best-case scenario where the over-trained model wins on both training and inference efficiency. It highlights how the conventional wisdom about "larger models being better" can be misleading for deployment contexts.

Even when over-training does cost more, break-even typically occurs at modest inference volumes. If the 7B model required $10^{24}$ FLOPs to train (an additional $8 \times 10^{23}$ FLOPs over the 70B option), the break-even would be:

$$T_{\text{break-even}} = \frac{8 \times 10^{23}}{1.26 \times 10^{11}} \approx 6.3 \times 10^{12} \text{ tokens}$$

At 500 tokens per request, that's about 12.6 billion inference requests, easily exceeded by any widely-deployed model. To put this in perspective, a popular API serving just 10,000 users, each making 10 requests per day, would reach this break-even point in roughly 3.4 years. For consumer-scale deployments with millions of users, break-even occurs within days or weeks of launch.