Power Laws in Deep Learning › Log-Log Linear Relationships

Understanding the mathematical form of power laws is only the beginning. In practice, a transformation helps you identify and validate power laws, then work with them by converting their curved relationships into straight lines. This transformation bridges theory and practice, turning a specialized nonlinear analysis into the familiar, well-understood territory of linear regression. If you understand only one technical tool from this chapter, let it be this one: the log-log transformation is the lens through which all scaling law analysis is conducted.

Taking the logarithm of both sides of $y = ax^b$ reveals the hidden linear structure. Let's step through this derivation carefully, as each step illuminates something important:

$$\begin{aligned} \log y &= \log(ax^b) \\ &= \log a + \log(x^b) \quad \text{(product rule: } \log(AB) = \log A + \log B \text{)} \\ &= \log a + b \log x \quad \text{(power rule: } \log(x^b) = b \log x \text{)} \end{aligned}$$

where:

• $\log y$: the logarithm of the dependent variable
• $\log a$: a constant term that becomes the y-intercept in the transformed space
• $b$: the power law exponent, which becomes the slope of the line
• $\log x$: the logarithm of the independent variable

To see the linear structure most clearly, introduce substitutions $Y = \log y$ and $X = \log x$. The equation becomes:

$$Y = (\log a) + b \cdot X$$

This is unmistakably the equation of a straight line $Y = mX + c$ in slope-intercept form, where the slope $m = b$ is the power law exponent and the intercept $c = \log a$ encodes the coefficient. Each component has a clear geometric interpretation:

• $\log y$: the logarithm of the dependent variable, which becomes our new vertical axis in the transformed space
• $\log a$: a constant term that becomes the y-intercept, the value where the line crosses the vertical axis when $\log x = 0$, that is, when $x = 1$
• $b$: the power law exponent, which becomes the slope of our line, telling us how many units $\log y$ changes for each unit change in $\log x$
• $\log x$: the logarithm of the independent variable, which becomes our new horizontal axis

Why does this matter so much? Because straight lines are easy to work with. We have centuries of well-developed tools for analyzing linear relationships, fitting them to data, and extrapolating from the result. By transforming power laws into lines, we gain access to this entire toolkit: linear regression, confidence intervals, residual analysis, hypothesis testing. The slope of a line is one of the most intuitive quantities in mathematics: it tells you the rate of change, and we have immediate intuitions about what different slopes look like and what they mean.

The slope interpretation is particularly useful for comparing scaling behaviors. If architecture A has a parameter-scaling slope of $-0.10$ in log-log space and architecture B has a slope of $-0.07$, architecture A's loss drops faster with increasing parameters. The difference between these slopes represents a 43% advantage in scaling efficiency (0.10 vs. 0.07) that compounds sharply over large scale ranges.

This transformation has three concrete practical implications that you'll use repeatedly when working with scaling data:

• Detection: If your data forms a straight line on log-log axes, where both axes are logarithmic, you've found a power law. This gives you a simple visual test. Plot your scaling data with logarithmic scales on both axes and look for linearity. No complex statistical test is needed at first.
• Parameter estimation: The slope gives you the exponent directly. You don't need specialized nonlinear optimization. Basic linear regression on log-changed data is enough and more reliable.
• Extrapolation: Linear extrapolation in log space corresponds to power law extrapolation in linear space. We can extend a straight line with confidence. The log-log transformation lets us translate that confidence to power law predictions across orders of magnitude beyond our observed data.

Let's make each of these steps concrete with a worked numerical example. Suppose loss $L$ decreases with compute $C$ according to:

$$L = 5.4 \cdot C^{-0.05}$$

where:

• $L$: the model's loss (lower is better, no units)
• $C$: compute budget in FLOPs (floating-point operations)
• $5.4$: the coefficient, representing the loss we would observe at $C = 1$ FLOP (an entirely hypothetical anchor point with no physical interpretation, but a useful mathematical reference)
• $-0.05$: the scaling exponent, with the negative sign showing loss decreases as compute increases

Before diving into the calculation, consider what this equation is telling us qualitatively. The coefficient 5.4 anchors the curve but isn't real on its own, since no real model trains with one floating-point operation. The exponent $-0.05$ carries the prediction: loss decreases slowly as compute increases, and the small magnitude (0.05) tells us the improvement per order of magnitude is modest. A large lab running this kind of experiment would typically see the curve covering 10 or more orders of magnitude of compute, from small ablation runs to the largest training runs they can afford.

Taking base-10 logarithms of both sides:

$$\begin{aligned} \log_{10} L &= \log_{10}(5.4 \cdot C^{-0.05}) \\ &= \log_{10}(5.4) + (-0.05) \log_{10} C \\ &= 0.73 - 0.05 \cdot \log_{10} C \end{aligned}$$

In log space, our line has y-intercept 0.73 and slope $-0.05$. Now we can answer a concrete question: if we increase compute by a factor of 10 (one order of magnitude), meaning $\log_{10} C$ increases by exactly 1, how much does loss change?

$$\Delta \log_{10} L = -0.05 \times 1 = -0.05$$

What does a change of $-0.05$ in $\log_{10} L$ mean for actual loss? We work backwards: if $\log_{10} L$ decreases by $0.05$, then $L$ is multiplied by:

$$10^{-0.05} \approx 0.891$$

The new loss is approximately 89.1% of its previous value. This is an 11% reduction in loss for every tenfold increase in compute. Each order of magnitude of compute reduces loss by about 11%. This fixed proportional improvement per order of magnitude is the signature of power law scaling, and it's the kind of prediction that makes planning possible.

Why does this formula make sense? Notice that the exponent $-0.05$ appears as a conversion factor between "orders of magnitude of compute" and "fractional reduction in loss." The negative sign enforces the inverse relationship. The magnitude (0.05) determines the efficiency of the conversion. If you had a larger exponent in magnitude, say $-0.10$, each order of magnitude of compute would buy you more improvement. Comparing scaling laws from different architectures or training configurations therefore amounts to comparing these exponents.

Let's extend this to a multi-step planning exercise. Suppose you have access to a compute budget that allows you to do a training run that costs $10^{21}$ FLOPs. You've run small experiments at $10^{17}$ and $10^{18}$ FLOPs, measured losses of approximately 3.0 and 2.8 respectively, and fit the power law above. What loss should you expect?

Step 1: Verify the fit. Using the formula $L = 5.4 \cdot C^{-0.05}$, compute predicted losses at your observed scales.

At $C = 10^{17}$: $L = 5.4 \cdot (10^{17})^{-0.05} = 5.4 \cdot 10^{-0.85} = 5.4 \cdot 0.141 \approx 0.76$.

Wait, that doesn't match the observed 3.0. This means our example power law $L = 5.4 \cdot C^{-0.05}$ needs re-anchoring to the observed data. Let's instead work with the observed data directly. From the two observations:

$$\begin{aligned} \log_{10}(3.0) &= \log_{10} a + b \cdot 17 \\ \log_{10}(2.8) &= \log_{10} a + b \cdot 18 \end{aligned}$$

Subtracting: $\log_{10}(2.8) - \log_{10}(3.0) = b \cdot 1$, so $b = \log_{10}(2.8/3.0) = \log_{10}(0.933) \approx -0.030$.

Then $\log_{10} a = \log_{10}(3.0) - (-0.030) \cdot 17 = 0.477 + 0.510 = 0.987$, so $a = 10^{0.987} \approx 9.7$.

At $C = 10^{21}$: $L = 9.7 \cdot (10^{21})^{-0.030} = 9.7 \cdot 10^{-0.63} = 9.7 \cdot 0.234 \approx 2.27$.

The predicted loss at your large training run is approximately 2.27. You've made a specific, quantitative, checkable prediction before spending a cent on the expensive run. This is the practical power of power law scaling.

Let's visualize this relationship concretely: