Kaplan Scaling Laws › Kaplan Scaling Laws

In January 2020, a team at OpenAI published findings that would change how the field thinks about training large language models. Their paper, "Scaling Laws for Neural Language Models," presented empirical evidence that language model performance follows predictable patterns as you increase model size, dataset size, and compute budget. These patterns, now known as the Kaplan scaling laws, provided a rigorous framework for predicting how much improvement you could expect from scaling up neural language models.

Building on our understanding of power laws from the previous chapter, we now examine how these mathematical relationships manifest specifically in language model training. The Kaplan team didn't just observe that performance improves with scale; they quantified the exponents governing these improvements and derived practical formulas for optimal resource allocation. Their central claim was clear: when you have a fixed compute budget, you should prioritize making models larger rather than training them on more data. This guidance shaped the development of GPT-3 and influenced the broader direction of large language model research.