Predicting Model Performance › Loss Extrapolation

The most direct application of scaling laws is predicting the test loss a model will achieve at a given scale. This prediction task asks a simple question: if we train a model with $N$ parameters on $D$ tokens of data, what loss should we expect? The answer comes from the mathematical relationships we've established between scale and performance.

From the Chinchilla scaling law we covered in Part XXIII, Chapter 3, we know that loss follows:

$$L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}$$

To understand what this formula captures, let's examine each component and the intuition behind the mathematical structure:

• $L(N, D)$: the predicted test loss for a model with $N$ parameters trained on $D$ tokens
• $N$: the number of model parameters (capturing model capacity)
• $D$: the number of training tokens (capturing data scale)
• $E$: the irreducible loss, the theoretical minimum achievable even with infinite compute, determined by the entropy of natural language
• $A$: the parameter scaling coefficient, controlling how much model size reduces loss
• $B$: the data scaling coefficient, controlling how much training data reduces loss
• $\alpha$: the parameter scaling exponent (typically ~0.34), determining how quickly returns diminish as models grow
• $\beta$: the data scaling exponent (typically ~0.28), determining how quickly returns diminish as data grows

The formula captures an important property: loss decomposes into three independent terms, each telling us something different about where performance comes from and what limits it. The irreducible loss $E$ sets a floor that no amount of scaling can breach. This represents the basic unpredictability in language itself, the entropy that remains even when we know all the patterns. If someone asks "What word comes next?", sometimes multiple words are plausible, and no model can do better than chance on these cases.

The power-law terms $A/N^{\alpha}$ and $B/D^{\beta}$ represent reducible error that diminishes as we scale, but with diminishing returns governed by the exponents. The mathematical form of power laws encodes a important property: each doubling of scale produces a constant fractional improvement rather than a constant absolute improvement. When $\alpha = 0.34$, doubling model size reduces the parameter-dependent loss term by a factor of $2^{0.34} \approx 1.27$, or roughly 21%. This same 21% reduction applies whether we're going from 1B to 2B parameters or from 100B to 200B. The absolute loss reduction shrinks, but the proportional improvement remains constant.

The additive structure, simply summing the three terms, reflects an important empirical finding: the contributions from model size and data size appear to be approximately independent. A larger model benefits equally from more data regardless of its starting size, and more data helps equally regardless of model capacity. This separability is what makes extrapolation tractable; if the terms interacted in complex ways, prediction would require understanding those interactions at scales we haven't tested.

Given these fitted parameters, extrapolating to larger scales seems straightforward: plug in your target $N$ and $D$, compute $L$. However, the practical challenge lies in fitting these parameters reliably from limited training runs. We need enough data points to constrain a five-parameter model, but we're typically trying to minimize how many expensive training runs we perform.

The standard approach trains a series of models at small scales, measures their final losses, then fits the scaling law parameters. The key insight is that the exponents $\alpha$ and $\beta$ appear stable across model families, so we primarily need to estimate the coefficients. This stability helps. It means that even if our small-scale experiments can't pin down the exponents exactly, we can borrow values from published research on similar architectures and focus our fitting effort on the coefficients that capture our specific training setup.

The fitting process treats our scaling law as a regression problem: we have observed data points (pairs of scale and achieved loss) and want to find parameters that best explain those observations. The nonlinear form of the power laws requires iterative optimization rather than closed-form solutions, but modern optimization libraries handle this efficiently.

With fitted parameters in hand, we can now predict performance at scales we haven't trained. This is where scaling laws earn their keep, transforming expensive experiments into inexpensive calculations. The prediction itself is just arithmetic: substitute our target values into the fitted formula. But understanding what we're doing mathematically helps us appreciate both the power and the limitations of this extrapolation.

When we plug in a target $N$ and $D$, we're asserting that the functional relationship we observed at small scales continues to hold at large scales. We're assuming that the same three-term structure, irreducible loss plus two power-law improvements, captures the physics of learning at 70 billion parameters just as well as at 70 million. This is a strong assumption, and one we can never fully verify without training the larger model.

These predictions extrapolate 1-2 orders of magnitude beyond our training data. The further we extrapolate, the more our predictions depend on the assumption that scaling relationships remain stable, an assumption we'll examine critically later in this chapter. This extrapolation is valuable: we use models that cost thousands of dollars to train in order to predict the behavior of models that cost millions of dollars. That cost difference explains why this technique has become central to AI research planning.

A visual representation helps build intuition for what extrapolation means geometrically. The scaling law defines a surface in three-dimensional space, where two dimensions represent our inputs (parameters and tokens) and the third dimension represents the output (loss). Our training data points lie somewhere on this surface, and fitting the scaling law means aligning the surface as closely as possible with them. Extrapolation asks: if we extend this surface beyond where we have data, what does it predict?