Power Laws in Deep Learning › Fitting Power Laws

To use power laws predictively, we need to estimate the parameters $a$ and $b$ from data. This is where the log-log transformation proves most useful in practice. Instead of wrestling with nonlinear curve fitting, which is sensitive to initialization, prone to local minima, and statistically tricky, we can use the simple, well-understood tools of linear regression on log-transformed data.

The strategy is straightforward: take logs of both the inputs and outputs, then apply ordinary linear regression. The slope of the resulting fit is the exponent $b$, and the intercept is $\log a$. This approach has several practical advantages over directly fitting $y = ax^b$ with nonlinear methods. Most importantly, it is numerically stable across the wide ranges typical of scaling experiments (compute budgets spanning 15 orders of magnitude, model sizes spanning 5 or more orders of magnitude). Nonlinear optimization applied directly to these ranges would require careful initialization and often fails to converge reliably.

Given data points $(x_i, y_i)$, we transform to $(\log x_i, \log y_i)$ and fit the linear model:

$$\log y = \log a + b \log x$$

where:

• $\log y$: the log-transformed dependent variable, computed from our measured loss values
• $\log x$: the log-transformed independent variable, computed from our resource measurements (parameters, data, compute)
• $\log a$: the y-intercept in log space, recovered from the linear regression
• $b$: the slope, which equals the power law exponent directly

Comparing this to the standard linear form $Y = mX + c$ makes the connection explicit. The slope $m$ in our transformed regression equals $b$. The intercept $c$ equals $\log a$, which we convert back to the original coefficient via exponentiation: $a = e^c$ if using natural logarithms, or $a = 10^c$ if using base-10 logarithms.

This approach works because ordinary least squares regression minimizes the sum of squared residuals. When we apply it to log-transformed data, we're effectively minimizing:

$$\sum_i \left( \log y_i - \log a - b \log x_i \right)^2 = \sum_i \left( \log \frac{y_i}{a x_i^b} \right)^2$$

This objective function penalizes relative errors, not absolute errors. A prediction that's off by 10% at loss 3.0 contributes the same squared residual as a prediction that's off by 10% at loss 1.0. This is exactly the right behavior when values span many orders of magnitude: relative errors are the real measure of fit quality.

Fitting power laws correctly requires care. Several subtle issues can trip up even experienced analysts, and understanding them will help you interpret published scaling results critically.

Use log-transformed data for regression, not raw data. This point deserves emphasis because it's counterintuitive. You might think, "I want to fit $y = ax^b$, so I should use nonlinear regression to minimize the sum of squared errors in $y$." The problem is that this approach overweights large values. If your losses range from 10 at small scales to 0.1 at large scales, the squared errors at the small end (potentially hundreds) will dwarf squared errors at the large end (fractions of 0.01). The optimizer focuses on matching the high-loss region while ignoring the low-loss region. By contrast, fitting in log space minimizes relative errors: a 10% miss at loss 10 contributes as much to the objective as a 10% miss at loss 0.1. This is usually what we want when values span many orders of magnitude.

Consider weighted regression for heteroskedastic data. If measurement uncertainty varies across your data range, weighted regression can improve estimates. For instance, if small-scale experiments are run once while large-scale experiments are run multiple times with averaged results, the large-scale points have lower variance and deserve more weight in the fit. In practice, this issue is common because large training runs are expensive and researchers typically replicate small-scale experiments more than large-scale ones.

Validate with residual analysis. After fitting, examine residuals in log space. If the residuals show systematic patterns, such as being consistently positive at small scales and negative at large scales, this indicates the power law model may not be appropriate. True power law data should produce residuals that scatter randomly around zero with no systematic trend. A curved residual pattern suggests you might be dealing with multiple regimes or a more complex functional form. This residual analysis step is often skipped in published work but is needed for building confidence in the fit.

Check for outliers before fitting. A single outlier in log space can have a disproportionate effect on the estimated slope. When fitting scaling laws, examine whether any data points deviate substantially from the trend before including them in the regression. Common sources of outliers include training runs that were stopped early, runs with suboptimal hyperparameters, or runs where something went wrong with the data pipeline.

Beware of the data selection effect. Researchers typically publish scaling experiments where the power law fit is clean. Experiments where the relationship is noisy or non-power-law are often not published. This means the published literature may systematically overestimate how cleanly power laws hold in practice. When you run your own scaling experiments, expect more noise than the published papers suggest.

Let's implement power law fitting and see these principles in action: