Compute-Optimal Training › Compute-Optimal Training

Scaling laws reveal how compute constrains model size and data. But knowing these relationships is only the first step. The practical question every training run must answer is: given a fixed compute budget, how should I allocate resources between model size and training tokens?

As we discussed in the Chinchilla Scaling Laws chapter, Hoffmann et al. demonstrated that earlier approaches from Kaplan et al. significantly underestimated the importance of training data. Chinchilla showed that compute-optimal training requires balancing model size and training tokens in roughly equal proportions. This chapter translates those theoretical findings into actionable guidelines for planning and executing training runs.

We'll derive the formulas for optimal resource allocation, work through concrete examples of budget planning, examine how training hyperparameters interact with compute efficiency, and establish practical recipes that practitioners can apply directly. By the end, you'll understand what compute-optimal training means and how to achieve it in practice.