Scaling Laws (2020) › The Problem

As language models grew larger and more sophisticated throughout the late 2010s, researchers faced a fundamental uncertainty: how should they allocate limited computational resources to maximize performance? The intuitive answer seemed to be scaling everything up, but without understanding the mathematical relationships between different scaling dimensions, decisions about resource allocation were essentially guesses. Should an organization invest in training a larger model, collecting more training data, or both? How much better would performance be if they doubled the model size? If they doubled the training data? If they doubled the computational budget allocated to training? Without answers to these questions, planning model development was fraught with risk.

Consider a scenario where a research team had a computational budget equivalent to training a 10 billion parameter model on 100 billion tokens. Should they train a 20 billion parameter model on 50 billion tokens? A 5 billion parameter model on 200 billion tokens? Or something else entirely? Without scaling laws, there was no principled way to answer these questions. Teams might default to making models as large as possible, assuming that bigger models would always perform better, but this assumption ignored the role of training data and the trade-offs between different scaling dimensions.

The lack of predictive scaling laws also made it difficult to estimate whether a training run would succeed before committing enormous computational resources. If a team wanted to achieve a specific performance target, how large should the model be? How much training data would be needed? Without scaling laws, these questions had no answers, forcing teams to rely on intuition, precedent from similar models, or trial and error. This uncertainty was particularly problematic given the enormous costs of training large models: a failed training run could waste weeks of compute time and thousands or millions of dollars in cloud computing costs.

The problem extended beyond individual training runs to strategic questions about the future of language model development. If researchers wanted to understand whether scaling could continue to improve performance indefinitely or whether they would eventually hit diminishing returns or fundamental limits, they needed systematic data about how performance scaled with resources. Without scaling laws, it was impossible to predict whether investing in larger models would continue to provide significant improvements or whether alternative approaches would be necessary to achieve further gains. This uncertainty made it difficult to plan long-term research directions or make strategic decisions about where to invest resources.

There was also a deeper theoretical problem: researchers needed to understand why and how scaling improved performance to develop better models and architectures. If they could identify the mathematical relationships governing how performance improved with scale, they might be able to design more efficient architectures that achieved better performance with fewer resources, or understand which aspects of scaling mattered most for different types of capabilities. Without scaling laws, these insights remained inaccessible, limiting the ability to improve models beyond simply making them larger.

The field needed systematic empirical investigation that would reveal how performance scales across different dimensions. This investigation would need to train many models across a wide range of scales, carefully controlling for different factors, and measuring how performance changed. The goal would be to discover whether simple mathematical relationships could accurately predict performance from scale alone, enabling researchers to make informed decisions about resource allocation and predict model capabilities before training began.