Power Laws in Deep Learning › Power Laws in Deep Learning

Throughout this book, we've built increasingly advanced language models, from simple n-gram models through BERT and GPT architectures to modern designs like LLaMA and Mistral. At each step, we made architectural choices: how many layers, how large a hidden dimension, how much training data to use. Most of those choices were made through intuition, ablation studies, or by following prior work. But a more basic question lurks beneath all of them: what governs model performance as we scale up? If we double the parameters, how much better does the model get? If we multiply the data tenfold, what happens to the loss?

For most of machine learning's history, these questions had no principled answer. Researchers knew bigger was often better, but "often" and "better" were vague. You could train a model with 100M parameters and then one with 1B parameters, and you could observe that the larger model performed better, but you couldn't predict by how much. Every large training run was a costly gamble. Teams at large research labs would commit enormous compute budgets to experiments whose outcomes were unknown, because they lacked a mathematical framework for forming an informed expectation. The result was a field that advanced partly through systematic experimentation, but also substantially through luck, intuition, and the willingness to spend money on gambles.

The answer, it turns out, lies in a deceptively simple mathematical relationship that governs neural network scaling: the power law. Power laws are not a new invention specific to deep learning. They appear in physics and economics, in biology and linguistics, and even in city population distributions. What makes them remarkable in the context of language models is how consistently and reliably they hold across many orders of magnitude of model scale. A relationship you observe at 10 million parameters predicts, with startling accuracy, what will happen at 100 billion parameters. This predictive power has transformed how teams approach large-scale model development, converting expensive experiments into measured extrapolations and turning resource allocation from guesswork into principled optimization.

Understanding power laws transforms how we think about training language models. Instead of guessing at the right model size or dataset, we can predict performance across orders of magnitude, make informed resource allocation decisions, and understand the basic constraints that govern what's possible with current approaches. Before you invest millions of dollars and months of compute time in a training run, you can run small-scale experiments, fit a power law, and read off a prediction for the large model's loss. The prediction is not perfect, but it's accurate enough to be useful. The ability to test an idea at 1% of the cost and then extrapolate reliably to full scale has changed what it means to do empirical research in large-scale ML.

Power laws also clarify the economics of model development in a way that informs strategic decisions. If the loss-versus-parameters relationship has a small exponent, doubling model size buys relatively little improvement, and you should think carefully about whether scaling is the right lever to pull. If the exponent is large, scaling is highly efficient, and additional investment in parameters or data will yield substantial returns. This framing moves scaling decisions from "spend as much as you can and hope" to something closer to quantitative investment analysis.

This chapter establishes the mathematical foundations of power laws, explores why they appear so universally in deep learning, and develops the intuition you'll need to interpret the scaling laws we'll examine in subsequent chapters. We'll move from the basic mathematical definition through the key log-log transformation trick, connect power laws to Zipf's law in language, and build practical skills for fitting and applying these relationships. By the end, you'll have the vocabulary and intuition to read scaling law papers with real understanding, not just familiarity with the formulas.