Power Laws in Deep Learning › Power Laws in Language

Power laws aren't new to language modeling. In fact, you've already encountered one of the most famous power laws in language earlier in this book: Zipf's law. This connection is more than historical curiosity. It reveals that power law behavior is woven into the very fabric of language itself, predating neural networks by decades and suggesting something deep about how natural language is structured. The fact that language itself exhibits power law structure is both a fascinating empirical observation and a clue about why language models might also scale according to power laws.

When we discussed word frequency distributions in the tokenization chapters, we noted that the frequency of words follows a remarkably consistent pattern. If you rank words by frequency, the $r$-th most common word appears with frequency proportional to $1/r$. Formally:

$$f(r) \propto \frac{1}{r^{\alpha}}$$

where:

• $f(r)$: the frequency of the word at rank $r$, meaning how many times it appears per million tokens
• $r$: the rank of the word (1 = most common, 2 = second most common, and so on)
• $\alpha$: Zipf's exponent, approximately 1 for most natural language corpora
• $\propto$: "proportional to," meaning the relationship holds up to a constant factor that depends on corpus size

The word "the" appears vastly more often than "cat," which appears vastly more often than "syzygy." This isn't a gradual decline. The most common words dominate text, while the long tail of rare words contributes relatively little to token counts. What's striking is that this distribution follows the same mathematical form across every language, every domain, every time period that linguists have studied. English novels, French newspapers, scientific papers, social media posts: all Zipfian. The universality is itself remarkable and suggests that Zipf's law shows something basic about how humans use language for communication, rather than being an artifact of any particular language or domain.

The exponent $\alpha \approx 1$ has a striking implication that we can trace through arithmetically. If the most common word appears 1 million times per million tokens (by definition, it accounts for 100% of tokens, which is impossible, so let's say it appears 70,000 times per million tokens as a more realistic estimate), then:

• Rank 10 word: $70,000 / 10^1 = 7,000$ occurrences
• Rank 100 word: $70,000 / 100^1 = 700$ occurrences
• Rank 1,000 word: $70,000 / 1000^1 = 70$ occurrences

Each factor of 10 in rank produces a factor of 10 decrease in frequency. This is scale invariance in action: the same proportional pattern repeating across the entire frequency spectrum. There is nothing special about rank 100 versus rank 1000: both follow the same law, and the law is the same whether you're looking at the top 10 words or the top 10 million words.

Zipf's law connects directly to the vocabulary problem we explored in the tokenization chapters. The heavy tail of rare words means that no matter how large your vocabulary, you'll encounter words not seen in training. This motivated subword tokenization approaches like BPE and WordPiece, which handle rare words by decomposing them into more frequent subunits. In this way, understanding the power law structure of language directly informed the design of modern tokenization systems. The mathematical structure we're studying now has practical engineering consequences that extend far beyond the abstract mathematical formalism.

There's an even deeper connection worth considering. If language itself exhibits power law structure, and language models are trained to model language, it's perhaps not so surprising that the models' learning dynamics also follow power laws. The statistical structure of the training data and the statistical structure of the model's performance improvements may be related at a basic level. A model trained to predict the next token in a Zipfian distribution faces a problem where common patterns are easy to learn and rare patterns are hard, and the difficulty of prediction tasks is itself distributed as a power law. The aggregate loss over this distribution, as the model improves, could plausibly follow a power law as a consequence.