Scaling Laws (2020) › 2020: Scaling Laws for Neural Language Models

In 2020, a team of researchers at OpenAI led by Jared Kaplan published findings that would shape how researchers and organizations approach the development of large language models. Their paper, "Scaling Laws for Neural Language Models," established mathematical relationships that predicted how model performance would improve as researchers increased model size, training data, or computational resources. These scaling laws revealed power-law relationships that enabled accurate predictions of model capabilities before training began. This provided central guidance for allocating resources and planning model development. The discovery that performance scaled predictably with these factors changed large language model development from an art into more of a science, giving researchers mathematical tools to make informed decisions about how to invest computational resources.

By 2020, the field had witnessed dramatic improvements from scaling transformer-based language models. GPT-2, released in 2019 with 1.5 billion parameters, had shown impressive capabilities. BERT, with its bidirectional architecture, had achieved strong performance across many NLP tasks. Researchers had observed that larger models generally performed better, but the relationship between scale and performance remained poorly understood. There was no systematic framework for predicting how much better a model would perform if given twice the parameters, twice the training data, or twice the compute. This uncertainty made it difficult to plan training runs, allocate resources efficiently, or understand the limits of scaling as a strategy for improving language models.

The lack of predictive scaling laws created significant problems for model development. Organizations planning to train large language models faced enormous uncertainty about what scale would be necessary to achieve target performance levels. Should they invest in a model with 10 billion parameters or 100 billion? How much training data would be needed? What computational resources would be required? Without answers to these questions, decisions about resource allocation were essentially guesses, leading to either wasted resources on over-engineered models or inadequate investment that failed to reach performance targets. The cost of training large models made these decisions particularly consequential, with failed training runs potentially wasting millions of dollars in compute time.

Kaplan and colleagues set out to systematically investigate how model performance scales with various factors. They trained transformer language models across a wide range of scales, varying model size from millions to billions of parameters, training data from millions to billions of tokens, and training compute accordingly. By measuring performance across these different configurations, they could identify mathematical relationships that predicted performance from scale alone. The goal was to discover whether simple power-law relationships could accurately predict how loss, and therefore performance, would improve with increased scale across different dimensions.

The results exceeded expectations. The researchers discovered clean power-law relationships between performance and scale across multiple dimensions. Model performance improved predictably with increased model size, with larger models achieving lower loss and better downstream task performance. Performance also improved predictably with increased training data, following similar power-law scaling. Most importantly, these relationships held across different model sizes and training configurations, suggesting that scaling laws were fundamental properties of neural language model training rather than artifacts of specific architectures or datasets. These laws enabled researchers to predict model performance before training, estimate resource requirements for target performance levels, and make informed decisions about how to allocate computational budgets across model size and training data.

The scaling laws had immediate practical consequences. They provided concrete guidance for planning training runs: researchers could now calculate approximately how many parameters and how much training data would be needed to reach a target loss, or predict how much better a model would perform if given additional resources. This predictive capability reduced uncertainty in model development and enabled more efficient resource allocation. The laws also revealed that performance improved smoothly with scale, suggesting that scaling was a viable long-term strategy for improving language models rather than hitting diminishing returns or fundamental limits.

The scaling laws also raised fundamental questions about the limits of scaling and the nature of intelligence that emerges from scale. If performance continued to improve smoothly with scale following power-law relationships, how far could this scaling continue? Would these relationships hold as models approached the limits of available training data or computational resources? The laws suggested that significant improvements were still possible through scaling, but they also hinted at potential limits and trade-offs that would become important as models grew larger. These questions would drive subsequent research, including the Chinchilla scaling laws that would refine understanding of compute-optimal training configurations.