Chinchilla Scaling Laws › Worked Example: Computing Chinchilla-Optimal Allocations

This section shows how to compute Chinchilla-optimal model sizes and dataset sizes for a given compute budget. This shows how the scaling equations translate into concrete numbers. Seeing the algebra work through a specific example is important because the abstract power-law relationships can feel distant from the practical decisions that model developers make. When you compute that a $10^{22}$ FLOP budget corresponds to a specific parameter count and token count, the equations stop being theoretical constructs and become engineering guidelines you can apply to your own training plans.

We will work through the calculation step by step, explaining the role of each factor and why the algebra takes the form it does. The derivation is short but worth following carefully, because understanding where the square root comes from and why the factor of 120 appears will help you apply the formula correctly in other contexts and catch errors when something does not look right.

Suppose we have a compute budget of $10^{22}$ FLOPs. Using the approximation that training FLOPs equals $6ND$ (where $N$ is parameters and $D$ is training tokens), and applying the Chinchilla ratio $D/N \approx 20$:

$$C = 6ND = 6N(20N) = 120N^2$$

where:

• $C$: the total compute budget in FLOPs
• $N$: the number of model parameters (what we want to solve for)
• $D = 20N$: substituting the Chinchilla-optimal token count
• The factor $120 = 6 \times 20$ combines the compute constant with the optimal ratio

This substitution works because if $D = 20N$ (the Chinchilla-optimal ratio), then the compute formula simplifies to a function of $N$ alone. This constrains us to move along the Chinchilla-optimal line in the parameter-token space and determines where along that line our compute budget places us. The quadratic relationship ($C \propto N^2$) emerges because both $N$ and $D$ grow together. Increasing $N$ automatically increases $D$ by the same factor, so compute grows as the square.

Solving for $N$:

$$N = \sqrt{\frac{C}{120}} = \sqrt{\frac{10^{22}}{120}} \approx 9.1 \times 10^9$$

where:

• $N$: the optimal number of model parameters we're solving for
• $C = 10^{22}$: our target compute budget in FLOPs
• The factor 120 comes from $6 \times 20 = 120$ (the compute constant 6 multiplied by the Chinchilla-optimal token ratio 20)
• The result $N \approx 9.1 \times 10^9$ means approximately 9 billion parameters

The square root appears because we're inverting the quadratic relationship: if $C = 120N^2$, then $N = \sqrt{C/120}$. This means that to double our optimal model size, we need to quadruple our compute budget. Similarly, a 10x increase in compute allows only about a 3.2x increase in model size (since $\sqrt{10} \approx 3.16$).

So approximately 9 billion parameters. The optimal token count is:

$$D = 20N = 20 \times 9.1 \times 10^9 \approx 1.82 \times 10^{11} \text{ tokens} \approx 182 \text{ billion tokens}$$

where:

• $D$: the optimal number of training tokens
• $N = 9.1 \times 10^9$: the optimal parameter count computed above
• The factor 20 is the Chinchilla-optimal tokens-per-parameter ratio

Let's verify the compute calculation:

$$C = 6ND = 6 \times (9.1 \times 10^9) \times (1.82 \times 10^{11}) \approx 10^{22} \text{ FLOPs}$$

This verification step confirms our arithmetic and shows that the Chinchilla-optimal allocation exactly consumes the available compute budget. Nothing is wasted. Every FLOP goes toward training a model that is neither too large (which would mean insufficient training) nor too small (which would mean wasted training iterations).

For comparison, the Kaplan scaling laws would suggest allocating more to model size. Using Kaplan's $N_{opt} \propto C^{0.73}$ and $D_{opt} \propto C^{0.27}$ (with appropriate constants), the same compute budget would suggest roughly 30-50 billion parameters trained on only 30-50 billion tokens. The Chinchilla allocation produces better loss for the same compute because the Kaplan allocation produces a severely undertrained model. With only about 1 token per parameter, the model has insufficient opportunity to learn from data, and many of its parameters remain poorly optimized.