Chinchilla Scaling Laws › The Chinchilla Scaling Equations

All three experimental approaches yielded consistent conclusions about optimal scaling. The Chinchilla team derived that for compute-optimal training, both model size and dataset size should scale as power laws of the compute budget. This convergence across three independent methods is the strongest form of empirical evidence: not only did the team find a result, they found it three times, using three different experimental designs, and it came out the same each time.

The mathematical form of these relationships is a power law, the same functional family that the Kaplan paper found. Such relationships also appear throughout physics and economics, as well as biology, when quantities scale with each other. A power law relationship $y \propto x^a$ says that when $x$ increases by a factor of 10, $y$ increases by a factor of $10^a$. If $a = 1$, they scale together linearly. If $a < 1$, $y$ grows more slowly than $x$. The scaling exponent $a$ therefore encodes the efficiency of converting one resource into another, and finding that exponent precisely is the core empirical contribution of the Chinchilla paper.

Before seeing the equations, notice an important structural constraint: compute $C$ is approximately equal to $6 \times N \times D$, where $N$ is parameter count and $D$ is training tokens. This means that if we increase $N$ and $D$ by some factors, compute increases by the product of those factors. Any claim about how $N_{opt}$ and $D_{opt}$ should grow with $C$ must be consistent with this multiplicative relationship. This constraint will force the scaling exponents $a$ and $b$ to satisfy a specific relationship, which we will work out below.

The Chinchilla team derived that for compute-optimal training, both model size and dataset size should scale as power laws of the compute budget:

$$N_{opt} \propto C^a$$ $$D_{opt} \propto C^b$$

where:

• $N_{opt}$: the optimal number of model parameters for a given compute budget
• $D_{opt}$: the optimal number of training tokens for a given compute budget
• $C$: the total compute budget (in FLOPs)
• $a$, $b$: scaling exponents that determine how parameters and data should grow with compute
• The constraint $a + b \approx 1$ follows from $C \propto N \times D$ (compute scales with the product of parameters and tokens)

These power law relationships mean that as your compute budget grows by some factor, both the optimal model size and dataset size grow by predictable amounts determined by the exponents $a$ and $b$. The proportionality symbol ($\propto$) indicates that while we don't specify the exact constants, the scaling relationship holds. If you increase compute by a factor of 100, the optimal parameter count increases by $100^a$ and the optimal token count increases by $100^b$.

The constraint that $a + b \approx 1$ is important because it comes from basic arithmetic rather than experiments. Since compute is approximately proportional to the product of parameters and tokens ($C \propto N \times D$), and since $N_{opt} \propto C^a$ and $D_{opt} \propto C^b$, we have $C \propto N_{opt} \times D_{opt} \propto C^a \times C^b = C^{a+b}$. For this equation to hold, we need $a + b = 1$. This constraint means that the exponents $a$ and $b$ cannot be chosen independently. They must sum to one, representing different ways of dividing the same computational pie.

The central finding was that $a \approx b \approx 0.5$, meaning model size and dataset size should grow at roughly the same rate with compute. This stands in stark contrast to the Kaplan scaling laws, which suggested $a \approx 0.73$ and $b \approx 0.27$, implying model size should grow much faster than dataset size.

Consider what this means in practice. With Chinchilla scaling (equal exponents around 0.5), if you increase your compute budget by a factor of 100, you should increase both your model size and your dataset size by a factor of 10 (since $100^{0.5} = 10$). With Kaplan scaling (unequal exponents), the same 100x compute increase would suggest increasing model size by about 50x ($100^{0.73} \approx 50$) but dataset size by only about 2x ($100^{0.27} \approx 2$). The difference matters: Kaplan scaling funnels most additional resources into model capacity, while Chinchilla scaling distributes resources evenly between capacity and training data.

Fitting the parametric loss model to experimental data yielded:

$$\alpha \approx 0.34, \quad \beta \approx 0.28$$

where:

• $\alpha$: the exponent governing how loss decreases with model size (from the $A/N^\alpha$ term)
• $\beta$: the exponent governing how loss decreases with training data (from the $B/D^\beta$ term)

These exponents determine how quickly each source of error decreases as we add parameters or data. The relatively similar values explain why balanced scaling is optimal: neither parameters nor data provide much faster diminishing returns compared to the other. If $\alpha$ were much larger than $\beta$, it would mean that increasing model size reduces loss much faster than increasing data, and we should favor parameters. If $\beta$ were much larger, we should favor data. But with $\alpha \approx 0.34$ and $\beta \approx 0.28$, both resources contribute roughly equally to loss reduction per unit of compute invested, so we should invest roughly equally in both.

The most practical finding from Chinchilla is the optimal ratio of training tokens to model parameters. The analysis suggests:

$$\frac{D_{opt}}{N_{opt}} \approx 20$$

where:

• $D_{opt}$: the optimal number of training tokens
• $N_{opt}$: the optimal number of model parameters
• The ratio of 20 emerges from the balanced scaling exponents ($a \approx b \approx 0.5$)

This means a compute-optimal model should be trained on approximately 20 tokens per parameter. A 1 billion parameter model should see roughly 20 billion tokens, a 10 billion parameter model should see 200 billion tokens, and so on.

This ratio provides a practical guideline: before training any model, you can estimate whether your training corpus is appropriately sized. Multiply your planned parameter count by 20, and that's roughly how many tokens you need for compute-optimal training. If you have fewer tokens available, you're training a model too large for your data. If you have many more tokens, you might benefit from a larger model.

This ratio is remarkably consistent across scales. The team found that deviating significantly from this ratio leads to compute inefficiency: either the model is too small to fully utilize the available training signal, or too large to be trained adequately given the data budget. The intuition is that each parameter represents a "slot" for storing learned knowledge, and 20 tokens provides enough gradient signal to fill that slot effectively. Too few tokens means the slot remains partially empty; too many tokens means the model cannot store all the patterns it could learn.