Compute-Optimal Training › Optimal Allocation Formulas

Let's derive the formulas for compute-optimal training from first principles. This derivation illuminates the mathematical machinery behind the Chinchilla recommendations and provides insight into why the specific ratios emerge. Building on the Chinchilla loss function:

$$L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta}$$

where:

• $L(N, D)$: the expected loss as a function of model size and data
• $E$: the irreducible entropy of natural language (the theoretical minimum loss)
• $A, B$: empirically fitted coefficients that determine the relative importance of model size vs. data
• $N^\alpha$: model parameters raised to power $\alpha \approx 0.34$, capturing how loss decreases with model size
• $D^\beta$: training tokens raised to power $\beta \approx 0.28$, capturing how loss decreases with more data

This functional form reveals important structure in language model training. The loss decomposes into three distinct components, each with a clear interpretation. The constant term $E$ represents the irreducible entropy of natural language, the inherent unpredictability that no model, however large or well-trained, can overcome. Human language has inherent randomness: there are many ways to phrase the same idea, and even expert human predictions of the next word would not achieve zero error. The second term $A/N^\alpha$ captures how loss decreases as you add model capacity. The power law form means that doubling your parameters doesn't halve this component of loss; rather, it reduces it by a factor of $2^\alpha \approx 1.27$. The diminishing returns are already built into this formulation. Similarly, the third term $B/D^\beta$ captures how loss decreases with more training data, again with diminishing returns governed by the exponent $\beta$.

The functional form reveals that loss decreases as power laws in both $N$ and $D$, but with different exponents, meaning model size and data have different marginal returns. To find the optimal allocation given a fixed compute budget $C = 6ND$, we minimize loss subject to this constraint.

Using the method of Lagrange multipliers (or substitution), the optimal allocation satisfies:

$$\frac{N_{\text{opt}}}{D_{\text{opt}}} = \frac{\beta}{\alpha} \cdot \frac{A}{B} \cdot \left(\frac{N_{\text{opt}}}{D_{\text{opt}}}\right)^{\alpha - \beta}$$

where all variables are as defined above for the loss function. This implicit equation emerges from setting the partial derivatives of the Lagrangian to zero. The left side represents the ratio of parameters to tokens we seek to determine. The right side involves the empirical constants from the loss function, weighted by the ratio of exponents. The self-referential nature of this equation, where the ratio appears on both sides, reflects the coupled nature of the optimization problem. It can be solved by substituting the empirically fitted values ($\alpha \approx 0.34$, $\beta \approx 0.28$, and the ratio $A/B$). After simplification, this yields a tokens-to-parameters ratio that Chinchilla estimated empirically:

$$\frac{D_{\text{opt}}}{N_{\text{opt}}} \approx 20$$

This ratio tells us that for every parameter in your model, you should train on approximately 20 tokens. A 1B parameter model should see about 20B tokens, and a 70B model should see about 1.4T tokens.

This result is notable for its simplicity. Despite the complexity of the optimization problem and the many factors that could influence training, the optimal ratio distills to a single number that holds across orders of magnitude in compute scale. This ratio is remarkably stable across compute scales. Whether you're training a 1B or 100B parameter model, compute-optimal training uses approximately 20 tokens per parameter.

To develop intuition for why 20 tokens per parameter is the magic number, consider what happens at the extremes. If you train with far fewer tokens per parameter (say 2), your model has enormous capacity relative to the training signal it receives. Many parameters will be undertrained, effectively storing noise rather than useful patterns. The model might memorize training examples rather than learning generalizable features. Conversely, if you train with far more tokens per parameter (say 200), you're repeatedly refining a model that has already extracted most of what it can from the data given its limited capacity. The marginal improvement from each additional token becomes negligible. The ratio of 20 represents the sweet spot where both forms of inefficiency are minimized.

Given a compute budget $C$ (in FLOPs), the compute-optimal model size and token count are:

$$N_{\text{opt}} = \sqrt{\frac{C}{6 \times 20}} = \sqrt{\frac{C}{120}}$$ $$D_{\text{opt}} = 20 \times N_{\text{opt}} = 20\sqrt{\frac{C}{120}}$$

These formulas follow from combining $C = 6ND$ with $D = 20N$. Substituting the second into the first gives $C = 6N \cdot 20N = 120N^2$, which we solve for $N$. This algebra turns the abstract scaling principles into concrete, calculator-ready equations. The factor of 120 in the denominator encapsulates both the computational cost structure (the 6) and the optimal data ratio (the 20), bundling all the Chinchilla insights into a single coefficient.

For back-of-the-envelope calculations, these can be approximated as:

$$N_{\text{opt}} \approx 0.091 \times \sqrt{C}$$ $$D_{\text{opt}} \approx 1.83 \times \sqrt{C}$$

where:

• $N_{\text{opt}}$: the optimal number of model parameters
• $D_{\text{opt}}$: the optimal number of training tokens
• $C$: total compute budget in FLOPs
• $0.091 \approx 1/\sqrt{120}$: derived from solving $C = 120N^2$ for $N$
• $1.83 \approx 20/\sqrt{120}$: the token coefficient, since $D = 20N$

These numerical coefficients make the formulas immediately usable. Given any compute budget expressed in FLOPs, you can quickly estimate the optimal allocation by taking the square root and scaling. The coefficient 0.091 for parameters is conveniently close to 0.1, making mental math tractable: a budget of $10^{22}$ FLOPs suggests optimal parameters around $0.1 \times 10^{11} = 10^{10}$, or roughly 10 billion parameters.

Let's implement these formulas: