Compute-Optimal Training › Beyond Compute Optimality

Compute-optimal allocation minimizes loss for a fixed compute budget, but this is not always the right objective. Several scenarios call for deviating from the Chinchilla ratio. Understanding when and why to deviate requires looking beyond "minimize validation loss given training compute" to consider the full lifecycle of a model.

If your model will serve many inference requests, a smaller, longer-trained model may be preferable. The extra training compute is a one-time cost, while inference compute accumulates over the model's lifetime. This asymmetry changes the optimization problem significantly.

Consider a model that will process 1 trillion tokens during deployment. The total lifetime compute is:

$$C_{\text{total}} = C_{\text{train}} + C_{\text{inference}} = 6ND + 2N \cdot T_{\text{inference}}$$

where:

• $C_{\text{total}}$: total compute over the model's lifetime (training plus inference)
• $C_{\text{train}} = 6ND$: training compute (forward and backward passes)
• $C_{\text{inference}} = 2NT_{\text{inference}}$: inference compute (forward passes only, hence factor of 2 not 6)
• $T_{\text{inference}}$: total tokens processed during inference deployment

The factor of 2 for inference (vs. 6 for training) reflects that inference only requires forward passes. During training, we compute the forward pass, then the backward pass for gradients, then apply optimizer updates. During inference, we only need the forward pass to generate predictions. This is why inference-heavy applications favor smaller, over-trained models. When $T_{\text{inference}}$ is large, the optimal training allocation shifts toward smaller models.

The implication is clear: for a model that will serve billions of inference requests, the optimal strategy may be to train a model that is two or three times smaller than the Chinchilla-optimal size, but train it on correspondingly more data. This "inference-optimal" model will have slightly higher loss than a Chinchilla-optimal model trained with the same compute, but the total lifetime cost (training plus inference) will be lower.

This reasoning led to models like LLaMA 2, which was 7B parameters trained on 2T tokens (286 tokens per parameter), intentionally "over-trained" by Chinchilla standards but more efficient at inference.

When high-quality training data is limited, you may not have enough tokens to train a compute-optimal model. In this case, you face a choice:

• Train a smaller model on available data (staying near 20:1 ratio)
• Train the larger model with data repetition (exceeding 1 epoch)
• Invest in data augmentation or synthetic data generation

We'll explore these tradeoffs in detail in the upcoming chapter on Data-Constrained Scaling.

Some capabilities emerge only above certain model sizes, as we'll discuss in Part XXX on Emergence. If your application requires a capability that emerges at 10B parameters, a compute-optimal 3B model will not suffice regardless of how much data you train on.

In such cases, the strategy is to train an undertrained larger model that crosses the capability threshold, then potentially continue training or fine-tune on domain-specific data.