Power Laws in Deep Learning › Practical Considerations in Power Law Analysis

When applying power law analysis to real neural network scaling data, several practical issues arise that pure theory doesn't address. Knowing these issues in advance helps you avoid common pitfalls and interpret results with appropriate confidence. The gap between textbook power law analysis and real-world scaling experiments is wider than it might appear from the clean figures in published papers.

Very small models may not follow the same power law as larger ones. There is often a "small-model regime" where performance is worse than the power law would predict. This makes sense intuitively. A model with 100 parameters cannot possibly learn the complex patterns that a transformer needs, regardless of what the power law formula suggests at that scale. The model is too small to represent the input data adequately, let alone generalize. Think of it as a minimum viable threshold: below it, the model doesn't even have enough capacity to attempt the task in a real way, and its performance shows architectural limitations rather than the statistical difficulty of learning.

The power law describes behavior in a regime where the model has enough capacity to engage in real learning. The threshold between "too small to fit the pattern" and "large enough to follow the power law" is an empirical question that varies with the task and architecture. When fitting scaling laws, it's common practice to exclude the smallest models from the fit to avoid contaminating the exponent estimate with finite-size effects. If you include them, the fitted exponent will be too small, because the performance improvement from the tiny regime to the power-law regime looks like a steeper slope than the underlying power law has.

Practically, this means you should be skeptical of power law predictions that extrapolate far below your smallest training point. If your smallest experiment used 10M parameters, predicting performance at 1M parameters using the fitted power law may be unreliable.

Power laws cannot continue forever. Eventually, models approach irreducible limits: noise in the data, inherent task ambiguity, the entropy of the underlying distribution, or the information-theoretic ceiling imposed by the task itself.

If the true labels in your dataset have inherent randomness, such as annotator disagreement on subjective tasks or fundamentally ambiguous examples, no model can reach zero loss. There is a floor below which you cannot go, regardless of how much you scale. In the language modeling literature, this floor is related to the entropy of human language, the irreducible uncertainty in predicting the next token given all prior context. The best possible language model could only reach perplexity equal to the true entropy of language; it could not do better.

When saturation effects are present, the observed scaling curve will deviate from the power law at large scales, bending upward and flattening toward the floor. If you fit a power law to data that includes the saturation regime, you'll underestimate the magnitude of the true exponent. The curve will appear to have a shallower slope than the underlying power law in the unsaturated regime, because the flattening at large scales pulls the slope downward.

One practical way to handle this is to use a modified power law model that incorporates an explicit floor. Instead of $L = aN^{-\alpha}$, use $L = L_{\min} + aN^{-\alpha}$, where $L_{\min}$ is the irreducible loss. This three-parameter model can be fit with slightly more data but gives a better characterization of scaling behavior near the saturation regime. The Kaplan et al. (2020) paper used exactly this form, estimating $L_{\min}$ for transformer language models on a specific text corpus.

Some scaling relationships show different power law exponents in different regions. For example, parameter scaling might have one exponent below 1B parameters and another above. These regime changes can correspond to qualitative shifts in what the model is learning.

One well-documented example involves emergent capabilities. For certain tasks, model performance appears flat or near-chance across a wide range of scales, then suddenly improves sharply over a narrow range. This phase transition-like behavior produces a scaling curve that doesn't fit a single power law. The flat region and the steep improvement region have very different effective exponents.

Multiple regimes also arise when the nature of the bottleneck changes with scale. A small model might be bottlenecked by its inability to stand for the needed features at all. A medium model might be bottlenecked by having too little data for its capacity. A large model might be bottlenecked by the information content of the training distribution. Each bottleneck has a different characteristic exponent. When you see a regime change in scaling data, the most productive question to ask is: "what bottleneck changed, and why did it change at this particular scale?"

Model size, data size, and compute are often increased together in practice. Separating their individual contributions requires careful experimental design. If you always double your data when you double your parameters, you cannot determine from the aggregate curve whether improvements come from more parameters, more data, or both. You're measuring a confounded combination of multiple power laws.

Controlled experiments that vary one factor while holding others constant are needed for understanding individual contributions. This is easier said than done at large scale, where a single training run consumes significant resources. Researchers address this by training a grid of models at different parameter counts and different data amounts, then fitting a joint model that separates the contributions of each. We'll see how this is done in the Chinchilla scaling laws chapter. The challenge is that the number of experiments required for a full factorial design grows multiplicatively with the number of factors, making exhaustive separation difficult at the frontier of compute.