Chinchilla Scaling Laws › The DeepMind Experiments

The Chinchilla paper is notable for its careful methods. Rather than relying on a single experimental approach, the DeepMind team used three independent methods to estimate optimal scaling, finding very consistent results across all three. This agreement between different approaches gives us strong confidence in the conclusions. When three different methods of measuring the same phenomenon all point to the same answer, we can trust that answer far more than if it came from a single experiment. We examine each approach in detail to understand both the methodology and the insights each reveals.

The scale of the experimental effort behind Chinchilla is worth appreciating. The team trained over 400 models in total, ranging from 70 million to 16 billion parameters. Each training run consumed real GPU hours and required careful coordination to ensure the results were comparable across configurations. This was not a theoretical analysis or a small proof-of-concept study: it was a systematic empirical program designed to answer a precise question about the optimal allocation of compute between model size and training data. That breadth, covering many model sizes and training durations rather than just a few flagship experiments, let the team test the allocation question across a wide range.

Understanding why the DeepMind team chose three separate methods, rather than just one definitive experiment, also illuminates something about the nature of scaling research. Each method captures a different slice of the data-generating process. The first method asks: given a model of fixed size, how does more data help? The second method asks: given a fixed compute budget, how should we split it between capacity and data? The third method asks: can we fit a mathematical model to all the data simultaneously and read off the optimal relationship analytically? The fact that all three converge on the same answer is not a coincidence but a deliberate experimental design choice intended to rule out artifacts from any single methodology.

The first approach trained over 400 models ranging from 70 million to 16 billion parameters. For each model size, the team varied the number of training tokens and measured the resulting loss. This produced loss curves as a function of training tokens for each model size.

The intuition behind this approach is straightforward: if we hold model size constant and simply vary how much data we train on, we can see exactly how each model size benefits from additional training. Some models might saturate quickly, suggesting they've learned all they can from the available data. Others might continue improving steadily, indicating they could benefit from even more training. By mapping these curves across many model sizes, we can compare how different architectures respond to different data quantities.

From these curves, the team extracted the minimum loss achievable at each compute budget. Here compute $C \approx 6ND$ for $N$ parameters and $D$ training tokens, and the factor of 6 accounts for both forward and backward passes across all operations. By examining which model size achieved the lowest loss at each compute level, they could determine the optimal allocation between parameters and data.

The key insight from this approach was that for any fixed compute budget, there exists an optimal model size, and training either too large or too small a model results in worse performance. Models that were too large couldn't be trained on enough data given the compute constraint, while models that were too small couldn't capture sufficient complexity regardless of training duration. This finding shows a key trade-off in neural network training: we must balance the model's capacity to represent complex patterns against its opportunity to learn those patterns from data. Neither capacity nor data alone determines performance. Their interaction does.

Notice that this approach requires a commitment to carefully controlled experiments. Simply reporting the loss after a fixed number of training steps would conflate two effects: the model's capacity and the amount of data it has seen. By explicitly varying training tokens for each model size and tracking the resulting loss curves, the team disentangled these factors. This careful separation is what makes the conclusions interpretable. When they later say that model X is "optimal" for compute budget C, they mean specifically: among all models tested, X achieves the lowest loss when given the tokens that its compute budget allows it to train on.

In practice, this means that the loss curves revealed something subtle: at any fixed compute level, the model that achieves the best loss is not necessarily the one that would eventually achieve the best loss if trained for much longer. A small model might reach excellent loss quickly and appear optimal at moderate compute, but a larger model, given the same total compute, might still be catching up. The IsoFLOP analysis in Approach 2 addresses this precisely, but the loss curve data from Approach 1 is what makes it possible to identify where the curves for different sizes cross as a function of total compute invested.

The second approach defined several fixed compute budgets (measured in FLOPs) and trained many models at each budget. For a given compute budget $C$, the team explored different allocations: some configurations used more parameters with fewer training tokens, while others used fewer parameters with more training tokens.

This approach inverts the question from the first method: rather than asking "how does performance vary as we give a fixed model more data?", it asks "given a fixed computational budget, how should we divide that budget between model capacity and training duration?" This framing directly addresses the practical question for any team with a finite compute budget: should we build a bigger model and train it less, or build a smaller model and train it more?

This created "iso-FLOP" curves showing how loss varies with model size at fixed compute. Each curve exhibited a clear minimum corresponding to the compute-optimal allocation. As the compute budget increased, this optimal point shifted to larger models, but importantly, the optimal dataset size grew at a similar rate. The shape of these curves, rising on both ends with a clear valley in the middle, visually demonstrates that extremes are costly: training a model that is too small wastes compute on redundant passes over data the model cannot fully use, while training a model that is too large wastes compute on parameters that never get enough gradient updates to learn useful representations.

The iso-FLOP methodology directly answers the question a practitioner faces when planning a training run. You cannot train everything: you have a GPU cluster, a time budget, and a target FLOPs count. Given that constraint, you need to decide how to configure your model. The iso-FLOP curves give you exactly that answer: find your budget on the x-axis, find the minimum of the corresponding curve, and read off the optimal model size. The fact that the curve has a clear, unambiguous minimum at every compute level tested means the answer is not sensitive to minor variations in experimental conditions.

The key insight from the iso-FLOP analysis is the shift in the optimal point as compute grows. If you plot the optimal model size at each compute budget, you get a curve that rises. If you plot the optimal token count, you get another curve that also rises. The question is: how fast does each curve rise relative to the other? This is exactly the question of whether the scaling exponents $a$ and $b$ are equal or unequal. The Chinchilla analysis found that both curves rise at approximately the same rate, meaning both the optimal parameter count and the optimal token count grow at the same pace as compute increases. This equal-rate growth is the mathematical foundation of the 20:1 token-to-parameter ratio and the practical prescription that model size and data should scale together.

The third approach fit a parametric function to predict loss as a function of both model size and training tokens:

$$L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta}$$

where:

• $L(N, D)$: the predicted loss for a model with $N$ parameters trained on $D$ tokens
• $N$: the number of model parameters
• $D$: the number of training tokens
• $E$: the irreducible entropy of natural language, the minimum loss achievable with infinite model size and infinite data
• $A$, $B$: learned scaling coefficients that determine how quickly loss decreases with more parameters or data
• $\alpha$, $\beta$: learned exponents that control the rate of diminishing returns for parameters and data respectively

This functional form provides a way to understand scaling behavior. Rather than treating each data point in isolation, this approach assumes that loss follows a specific mathematical structure, then fits that structure to all observed data simultaneously. The advantage is that once we have reliable estimates of the parameters, we can extrapolate to compute regimes we haven't yet explored and predict what loss we might achieve with a model ten times larger than any we've trained.

This functional form captures three contributions to the loss:

• $E$ represents the irreducible entropy of natural language, the minimum loss achievable with infinite model size and infinite data. This term sets a floor that no amount of scaling can overcome. Even a perfect model with unlimited training cannot predict language better than the inherent randomness in human text allows. This irreducible component reflects the unpredictability of language. sometimes multiple valid words could follow a given context, and no model can know which one a human will choose.
• $\frac{A}{N^\alpha}$ captures how loss decreases as model capacity increases. The power law form reflects diminishing returns. Each doubling of parameters yields a fixed percentage improvement, not a fixed absolute improvement. Moving from 1 billion to 2 billion parameters provides the same relative gain as moving from 100 billion to 200 billion parameters. The exponent $\alpha$ controls how steep this diminishing return curve is. Larger values of $\alpha$ mean faster initial improvement but quicker saturation.
• $\frac{B}{D^\beta}$ captures how loss decreases as training data increases. Similarly, this term shows that more data helps, but with diminishing returns governed by the exponent $\beta$. The first million tokens teach the model basic language structure. The next million refine that understanding. Eventually, additional tokens provide only marginal improvements as the model has already captured most learnable patterns.

By fitting this function to the experimental data, the team estimated the exponents $\alpha$ and $\beta$ along with the constants $A$, $B$, and $E$. These parameters then allow computation of the optimal allocation for any compute budget. This approach has predictive power: rather than running expensive experiments at every scale we care about, we can fit the function to data from smaller, cheaper experiments and then mathematically derive what should happen at larger scales.