Power Laws in Deep Learning › Building Power Law Intuition

Understanding power laws conceptually helps you reason about scaling decisions without constantly returning to the equations. Mathematical formulas are needed, but developing intuition about what those formulas mean allows you to think fluently about scaling in conversation, when reading papers, or when making resource allocation decisions. Here are the key intuitions to internalize.

The most important shift in perspective that power law thinking demands is moving from absolute quantities to ratios and proportions. When you think about linear relationships, you naturally think about how much you're adding: "we added 10 billion parameters." When you think about power law relationships, you should think about the ratio: "we multiplied our parameter count by 10." The absolute quantity is almost never the needed one. The ratio is what the power law responds to, and training your intuition to think in ratios is the single most practical habit you can develop for scaling law work.

A power law of the form $L = aN^{-\alpha}$ with $\alpha > 0$ means each additional parameter contributes less than the previous one, but never zero. Let's make each term explicit:

• $L$: the loss we're trying to minimize
• $N$: the number of model parameters
• $a$: a constant that sets the overall scale
• $\alpha$: the positive scaling exponent, with $\alpha > 0$ so diminishing returns

The negative exponent ensures that as $N$ increases, $L$ decreases, but the rate of decrease slows as $N$ grows larger. The millionth parameter is worth less than the first, but it's not worthless. This gradual decline contrasts sharply with exponential decay, where returns plummet rapidly after an initial period. Power law diminishing returns are persistent. There is always some benefit to scaling further, even if the marginal gain shrinks.

Think of it as a highway that has no speed limit but where the speed of traffic (your loss improvement) follows a known curve. You can always go faster by adding more lanes (parameters), but each additional lane contributes a bit less than the previous one. You never reach a point where additional lanes are completely useless. You just get diminishing returns forever. This means that the question "how much should we scale?" has no absolute answer based on the power law alone. It depends on your cost function: how much compute you can afford, how much improvement is worth paying for, and what alternative uses exist for your resources.

Power law thinking is multiplicative rather than additive. Going from 1B to 2B parameters gives the same proportional improvement as going from 10B to 20B parameters. In both cases, you're doubling the parameter count, and power laws respond to proportional changes, not absolute ones.

When planning scale-ups, think in terms of 2$\times$, 10$\times$, or 100$\times$ increases rather than adding fixed numbers of parameters. Asking "what if we add another billion parameters?" uses the wrong framing. It describes an absolute change that becomes increasingly irrelevant as models grow larger. Asking "what if we double our model size?" aligns with power-law behavior and produces consistent predictions regardless of starting scale. A team planning to scale from 10B to 20B parameters has made the same proportional commitment as a team scaling from 1B to 2B, and the power law predicts the same proportional improvement in both cases.

This multiplicative mindset extends to how you think about data and compute. If you have a 100B token dataset and consider doubling it to 200B tokens, power law thinking says: what exponent governs data scaling, and what does a 2$\times$ increase in data buy us? The answer to that question is the same whether you're going from 100B to 200B tokens or from 1T to 2T tokens. The proportional change is identical, and power laws respond to proportional changes.

The multiplicative framing also clarifies comparisons between different resource types. If doubling model size reduces loss by 5% and doubling data reduces loss by 7%, you should prefer data scaling, all else being equal. But you need to think about what doubling each resource costs, because the cost of doubling parameters (manufacturing more compute, memory, communication bandwidth) and the cost of doubling data (crawling, filtering, tokenizing more text) are very different. The power law tells you the return; cost accounting tells you the investment; together they tell you where to allocate resources.

The exponent is the most important number in any power law for a given scaling regime. Small differences in exponents compound into large differences at scale.

Consider two models with different parameter scaling exponents. If $L \propto N^{-0.07}$, doubling parameters multiplies loss by:

$$2^{-0.07} \approx 0.953$$

This is a 4.7% reduction. If instead $L \propto N^{-0.1}$, doubling parameters multiplies loss by:

$$2^{-0.1} \approx 0.933$$

This is a 6.7% reduction. A difference of 0.03 in the exponent might seem minor, but these differences compound sharply over large scale ranges. Over a 1000$\times$ scale-up:

• Exponent $-0.07$: loss becomes $1000^{-0.07} \approx 0.63$, a 37% reduction
• Exponent $-0.10$: loss becomes $1000^{-0.1} \approx 0.50$, a 50% reduction

A difference of 0.03 in exponent turns into a 13 percentage point difference in total improvement over three orders of magnitude of scaling. If you're planning a training run that spans six orders of magnitude, the difference would be even more large. Understanding your specific power law exponent is important for efficient resource allocation.

The exponent is also the quantity that distinguishes architectures. When researchers claim that architecture A scales better than architecture B, what they often mean is that A has a larger-magnitude exponent: a given proportional increase in parameters buys more improvement for A than for B. The coefficient $a$ matters less, because the coefficient affects where you start, but the exponent determines where you end up after scaling. Two architectures with the same coefficient but different exponents will produce different models after a 1000$\times$ scale-up, even if they started from the same loss. The architecture with the larger exponent will win decisively at large scale even if it's slightly worse at small scale.

This has a concrete implication for architecture search: when evaluating a new architectural idea, measure its absolute performance at a convenient scale and how its scaling exponent compares to the baseline. An architectural change that looks promising at small scale may have a slightly worse exponent and underperform at large scale. Conversely, an architectural change that looks modest or even slightly harmful at small scale may have a better exponent and sharply outperform at large scale.

Power laws make prediction possible in a way that arbitrary curves don't. Unlike a function specified by many parameters, a power law is fully specified by just two numbers: $a$ and $b$. If you measure performance at two scales, you have enough information to determine both parameters and therefore the entire curve.

In practice, you'd want more than two points to get a reliable estimate with confidence intervals. But the main point is that small-scale experiments inform large-scale predictions. If you train a 100M parameter model and a 1B parameter model, you can fit a power law and predict performance for a 100B parameter model. That's extrapolating 100$\times$ beyond your largest observation, based on just two data points. The prediction is not exact, but in the literature, researchers have been surprised by how accurate these extrapolations turn out to be.

This predictive power lets rational planning for expensive training runs. Instead of committing to a 1B parameter model and hoping for the best, you can train a series of models at 10M, 30M, 100M, and 300M parameters, fit a power law, and read off the predicted performance at 1B. If the prediction is promising, you proceed. If not, you investigate whether your architecture or data pipeline can improve the exponent before committing to the large run. This iterative "measure small, predict large, decide" workflow has become standard practice at research labs doing large-scale training.

The workflow also applies to intermediate-scale decisions. Should you train at 1B parameters or 3B parameters? If your power law predicts that 3B will reduce loss by an additional 7% compared to 1B, and you've already determined that this 7% improvement translates to a real performance gain on your downstream task, the decision is quantitatively grounded rather than intuitive. The power law is the bridge between the cost you pay (3$\times$ more compute) and the benefit you receive (7% lower loss).