Kaplan Scaling Laws › Summary

The Kaplan scaling laws showed that language model performance follows predictable power-law relationships across three dimensions:

• Loss vs. parameters: $L(N) \propto N^{-0.076}$, meaning doubling model size reduces loss by about 5%
• Loss vs. data: $L(D) \propto D^{-0.095}$, with slightly steeper improvement per unit of data
• Loss vs. compute: $L(C) \propto C^{-0.050}$, assuming optimal allocation

To interpret these exponents: a 10x increase in any factor reduces loss by $10^\alpha$, so parameters give $10^{0.076} \approx 1.19\times$ improvement, data gives $10^{0.095} \approx 1.24\times$, and compute gives $10^{0.050} \approx 1.12\times$ improvement.

The key practical finding was the optimal allocation formula: when scaling up compute, increase model size as $N \propto C^{0.73}$ and data as $D \propto C^{0.27}$. This prescription (scale models faster than data) shaped the development of GPT-3 and showed that larger models are more sample-efficient.

These relationships held cleanly across four orders of magnitude in the original experiments, allowing extrapolation to scales that hadn't yet been trained. However, the methodology for determining optimal allocation had systematic limitations that later work would revise. The next chapter examines the Chinchilla scaling laws, which challenged Kaplan's recommendations and showed that models like GPT-3 were undertrained relative to their size.