Compute-Optimal Training › The Compute Budget Problem

Training large language models is fundamentally a resource allocation problem that shares structural similarities with classical economics. Just as a business must decide how to allocate a fixed budget between labor and capital equipment, a machine learning practitioner must decide how to allocate a fixed compute budget between model capacity and training exposure. This decision is not just technical. It determines the fundamental character of the resulting model and shapes every downstream consideration from inference costs to capability profiles.

You have a fixed compute budget, typically measured in FLOPs (floating-point operations), and you must decide how to spend it. The two primary levers are:

• Model size ($N$): More parameters mean more expressive capacity but also more FLOPs per training step
• Training tokens ($D$): More data exposure improves learning but requires more total compute

Understanding why these two dimensions matter requires thinking about what happens during training. Model parameters represent the capacity of your neural network to store patterns and abstractions learned from data. A larger model can represent finer distinctions and capture longer-range dependencies. However, this capacity is merely potential. It must be actualized through exposure to training data. Training tokens represent the raw material from which the model extracts these patterns. Each token processed during training provides gradient signals that sculpt the parameter space, pushing the model toward configurations that predict text well.

The relationship between these quantities and total compute follows the approximation we established in earlier chapters:

$$C \approx 6ND$$

where:

• $C$: total compute in FLOPs (floating-point operations)
• $N$: the number of non-embedding parameters in the model
• $D$: the number of training tokens processed
• The factor of 6: accounts for approximately 2 FLOPs per parameter in the forward pass and 4 FLOPs per parameter in the backward pass (computing gradients and updating weights)

This formula deserves careful unpacking because it encapsulates a fundamental constraint. The factor of 6 arises from the computational structure of neural network training. During the forward pass, each parameter participates in roughly 2 floating-point operations (a multiply and an add for weighted sums). The backward pass requires computing gradients, which involves traversing the computational graph in reverse, costing approximately 2 more FLOPs per parameter. Finally, the optimizer update itself, when applying the gradient to modify weights, adds another 2 FLOPs per parameter. Together, these operations yield the factor of 6. This means that every token you process incurs a computational cost proportional to your model size, and every parameter you add increases the cost of processing every token.

Given this constraint, doubling your model size while keeping compute constant requires halving your training tokens. Conversely, doubling training tokens means using a smaller model. The question is: which allocation produces the best final model?

The Chinchilla findings established that compute-optimal training scales parameters and tokens equally with the compute budget. This result changes the interpretation for how we think about model development. It tells us that neither model capacity nor data exposure is inherently more valuable; they are complements that must grow in tandem.

$$N_{\text{opt}} \propto C^{0.5}$$ $$D_{\text{opt}} \propto C^{0.5}$$

where:

• $N_{\text{opt}}$: the optimal number of model parameters for a given compute budget
• $D_{\text{opt}}$: the optimal number of training tokens for a given compute budget
• $C^{0.5}$: the square root of compute, indicating both quantities scale with the same exponent
• $\propto$: indicates proportionality (the quantities scale together, differing only by a constant factor)

The exponent of 0.5 appearing for both quantities is the mathematical signature of balanced scaling. To understand why this matters, consider what the square root implies operationally. When you increase your compute budget by a factor of 10, you should increase both model size and training tokens by a factor of $\sqrt{10} \approx 3.16$. Neither dimension dominates, and both grow at the same rate.