Scaling Frontiers › Beyond Power Laws: When Scaling Breaks Down

The power law relationship between compute and performance is not a law of nature. It is an empirical observation that has held across a range of scales studied so far. Understanding when and why it might break down helps us anticipate what comes next.

One reason power laws have held so reliably is that scale and algorithmic improvements have historically been correlated. When researchers train larger models, they also tend to adopt better architectures, better optimizers, and better data curation pipelines. Separating the contribution of scale from the contribution of these other improvements is difficult in practice.

This correlation creates what we might call an overhang effect: the measured scaling exponents may overestimate the contribution of raw parameter count, because the experiments that establish those exponents also benefit from accumulated algorithmic improvements. If algorithmic progress slows, the empirical scaling curves may steepen, meaning each doubling of parameters delivers less improvement than historical data suggests.

Conversely, if a major architectural improvement arrives, it can create a step change in the scaling curve. The transition from recurrent architectures to transformers, which we covered in Parts H and I, was exactly this kind of shift. The transformer did not simply add more compute efficiency. It enabled a qualitatively different scaling regime by allowing much more parallelism during training and much more effective use of context.

The Chinchilla paper from DeepMind in 2022 fundamentally revised the community's understanding of optimal scaling. Prior to Chinchilla, the dominant intuition, influenced by the original Kaplan scaling laws, was that model size was the primary lever and that more parameters were always better, even if this meant training on proportionally less data.

Chinchilla showed this was wrong. For a given compute budget $C$, the optimal strategy allocates roughly equal resources to model size $N$ and training tokens $D$:

$$N_{\text{opt}} \propto C^{0.5}, \quad D_{\text{opt}} \propto C^{0.5}$$

This changed more than the numerical estimates. It revealed that many existing models were significantly undertrained relative to their size. GPT-3, with 175 billion parameters, was trained on roughly 300 billion tokens. A Chinchilla-optimal model at the same compute budget would use around 67 billion parameters trained on 1.4 trillion tokens.

The Chinchilla correction changes the interpretation of scaling results. If prior models were undertrained, then scaling up model size without proportionally scaling data was leaving performance on the table. This means the apparent "scaling plateau" observed in some early large models was partly an artifact of suboptimal data-to-parameter ratios, not a fundamental limit.

However, the Chinchilla prescription itself becomes harder to satisfy as models grow. Training a compute-optimal model at $10^{26}$ FLOPs would require roughly $10^{13}$ training tokens, which is orders of magnitude more than all indexed text on the internet. This is where the data wall enters.

A more subtle problem with relying on power law extrapolation is that the exponents themselves may shift at larger scales. The power law fits we observe are typically derived from experiments spanning at most two to three orders of magnitude. Extrapolating across five or ten orders of magnitude assumes the relationship is stable, but there is no theoretical guarantee of this.

Empirically, we have seen cases where anticipated performance improvements from scaling did not materialize as expected. The "GPT-4 moment" was real, but subsequent scaling from similar architectures has reportedly shown diminishing surprise. This may reflect a real departure from the law, or it may reflect that we have moved into a regime where other bottlenecks dominate.

The key insight is that power laws describe scaling in a single bottleneck regime. When the bottleneck shifts, whether from model capacity to data quality to alignment with human preferences to inference-time reasoning, the scaling curve can change its slope or even its functional form.