Power Laws in Deep Learning › What is a Power Law?

Before we can understand how neural networks scale, we need to establish the mathematical vocabulary that describes scaling relationships. The concept we're after is the power law, a particular type of functional relationship that, as you'll soon see, captures something needed about how complex systems behave as they grow. The power law is a mathematical framework for expressing empirical scaling laws, and fluency with it is a prerequisite for understanding every major result in the scaling laws literature.

A power law describes a relationship where one quantity varies as a power of another. This might sound abstract at first, so let's ground it with an intuition. Think of a power law as the mathematical description of how "bang for your buck" changes with scale. When you're small, every additional resource buys a lot of improvement. As you grow larger, each additional resource buys less improvement, but the relationship between resources and improvement follows a predictable, consistent pattern. The power law makes that pattern precise. It's like the difference between a vague promise ("you'll get better with practice") and a quantitative guarantee ("for every tenfold increase in practice hours, your error rate drops by 30%"). The power law gives the quantitative guarantee.

Imagine you're measuring how some output changes as you adjust an input. In the simplest case of a linear relationship, doubling the input doubles the output. The world of power laws is different and more interesting. If you double the input, the output doesn't simply double. Instead, it changes by a fixed multiplicative factor that depends on a special number called the exponent. This exponent is the heart of a power law: it tells you exactly how sensitive the output is to changes in the input. If the exponent is $-0.1$, doubling the input multiplies the output by $2^{-0.1} \approx 0.933$. If the exponent is $-0.5$, doubling the input multiplies the output by $2^{-0.5} \approx 0.707$. The magnitude of the exponent directly governs how quickly the relationship changes.

The general form is:

$$y = ax^b$$

where:

• $y$: the dependent variable (output quantity). This is what we're measuring or predicting. In deep learning contexts, it's often the loss of our model.
• $x$: the independent variable (input quantity). This is what we control or vary: parameters, data, compute, or training steps.
• $a$: the coefficient, which sets the overall scale of the relationship. Think of $a$ as calibrating where the curve sits vertically. If $a$ is large, the entire curve shifts upward; if $a$ is small, it shifts downward. The coefficient captures baseline performance characteristics specific to your architecture, task, or data. The coefficient $a$ tells you the value of $y$ when $x = 1$, since $a \cdot 1^b = a$ for any $b$.
• $b$: the exponent (or scaling exponent), which determines how steeply $y$ changes as $x$ increases. This is the most important parameter for understanding scaling behavior. The exponent tells us whether improvements come quickly or slowly as we add resources.

When $b$ is negative, we often write the relationship as:

$$y = \frac{a}{x^{\alpha}}$$

where $\alpha = -b > 0$ is the positive scaling exponent. This alternative notation proves convenient in deep learning contexts because it makes the relationship more immediately interpretable. When written this way, we can see at a glance that as $x$ increases, $y$ decreases, which matches our intuition that adding more parameters or data should improve, that is, lower, our loss. The positive exponent $\alpha$ directly tells us the rate of this improvement. A larger $\alpha$ means faster improvement; a smaller $\alpha$ means slower improvement. In the scaling laws literature, you'll frequently see expressions like $L \propto N^{-0.076}$ or $L \propto C^{-0.050}$, where $N$ is parameter count, $C$ is compute, and the reported number is the positive exponent $\alpha$.

Now that we have the basic form, let's understand what makes power laws distinctive compared to other mathematical relationships you might encounter. The key distinction is in how each type of function responds to proportional changes in the input. Consider three common functional forms and how they behave as we scale $x$ from small to large:

Linear: $y = ax + c$. Doubling $x$ adds a fixed amount to $y$. If you go from 100 to 200 parameters, you get the same absolute improvement as going from 1 billion to 1 billion and 100 parameters. This kind of relationship treats all additions equally, regardless of scale. However, this doesn't match how neural networks behave: there are clearly regions of diminishing returns. Adding one neuron to a 10-neuron network is a much bigger deal than adding one neuron to a billion-neuron network.

Exponential: $y = a \cdot e^{bx}$. Doubling $x$ multiplies $y$ by a fixed factor that grows explosively. Small changes in $x$ can produce enormous changes in $y$. Exponential relationships describe radioactive decay, compound interest, and viral spread, situations where growth feeds on itself. Neural network scaling doesn't show this kind of explosive behavior. If performance scaled exponentially with parameters, a 2$\times$ increase in model size would produce the same proportional benefit as any other 2$\times$ increase, regardless of the starting scale. We don't observe this.

Power law: $y = ax^b$. Multiplying $x$ by any factor $k$ multiplies $y$ by $k^b$. The key insight is that power laws respond to proportional changes, not absolute changes. Whether you're scaling from 10 million to 100 million parameters or from 10 billion to 100 billion parameters, the proportional change is the same (a factor of 10), and therefore the proportional effect on the output is the same.

Think of it this way: a linear relationship is like a salary increase where you always get $10,000 more per year of experience. An exponential relationship is like compound interest, where each year's earnings are reinvested. A power law is like a negotiated raise where each additional year of experience buys a fixed percentage improvement in salary, but that percentage shrinks as you become more experienced. The shrinking-percentage character of power law returns is exactly what we observe in neural network scaling: each additional order of magnitude of compute or parameters helps, but by a bit less than the previous order of magnitude, and the reduction follows a completely predictable mathematical rule.

The distinctive feature of power laws is scale invariance: the relationship between proportional changes stays constant regardless of where you are on the curve. A 10$\times$ increase in model parameters produces the same proportional decrease in loss whether you're going from 100M to 1B parameters or from 1B to 10B parameters. This property matters because it means power laws look the same at every scale.

Think of scale invariance like a fractal: zoom in or zoom out, and the mathematical structure is unchanged. If you look at the power law relationship between compute and loss from one vantage point, doubling your compute buys you the same percentage improvement as doubling your compute from any other vantage point. This self-similarity at all scales is not just mathematically elegant. It is the foundation of practical predictability.

Scale invariance is what lets us to make predictions about enormous models based on experiments with much smaller ones. When we run a series of experiments from 1M to 100M parameters, we're not just learning about small models. We're measuring the exponent of a power law that will continue to hold at 100B parameters. The small experiments are effectively sampling from the same underlying mathematical structure that governs the large ones. This is a remarkable fact: it means that studying a 10-million parameter model tells you something quantitatively real about a 100-billion parameter model, even though those models differ by a factor of 10,000 in size and are almost certainly learning qualitatively different internal representations.

Why does this property hold? A formal characterization is that power laws are the only functions that are scale-invariant in the following sense: if $f(kx) = g(k) \cdot f(x)$ for all values of $k$ and $x$, then $f$ must be a power law. No other functional form satisfies this multiplicative scaling property exactly. This makes power laws mathematically special, not just empirically convenient. The requirement is that proportional changes in the input always produce proportional changes in the output, with the proportionality factor depending only on the scale factor $k$ and not on the absolute value of $x$. Proving that power laws are the unique family satisfying this condition is a straightforward exercise in functional equations, but the conceptual point is more important: scale invariance is not a coincidence. It is the defining property of power laws, and any relationship we observe in nature that turns out to be scale-invariant will necessarily be a power law.