Scaling Laws (2020) › The Solution

Kaplan and colleagues addressed these problems through systematic empirical investigation that trained transformer language models across a wide range of scales and measured how performance scaled with model size, training data, and computational resources. The solution involved training models with sizes ranging from 768 thousand to 1.5 billion parameters, training them on datasets ranging from 22 million to 23 billion tokens, and systematically varying model width and depth alongside training compute. By measuring cross-entropy loss on held-out test data across all these configurations, the researchers could identify mathematical relationships that predicted performance from scale alone.

The key insight emerged from analyzing how loss scaled with different factors. The researchers discovered that test loss followed power-law relationships with model size, dataset size, and compute. Specifically, they found that test loss decreased predictably as a power-law function of model parameters, training data, and computational resources. These relationships held across different model configurations and training setups, suggesting that power-law scaling was a fundamental property of neural language model training rather than an artifact of specific choices.

The researchers formulated these relationships mathematically. They found that test loss $L$ followed power-law relationships with model size $N$, dataset size $D$, and training compute $C$. Specifically, they discovered that loss decreased with model size as $L \propto N^{-\alpha_N}$ for some scaling exponent $\alpha_N$, with dataset size as $L \propto D^{-\alpha_D}$ for exponent $\alpha_D$, and with compute as $L \propto C^{-\alpha_C}$ for exponent $\alpha_C$. These relationships enabled researchers to predict how much performance would improve with increased scale in any dimension.

The scaling laws also revealed important interactions between different scaling dimensions. The researchers found that optimal performance for a given computational budget required balancing model size and training data appropriately. If too much of the computational budget was allocated to model size with insufficient training data, the model would be undertrained. If too much was allocated to training data with too small a model, the model would lack sufficient capacity. The scaling laws provided guidance for finding optimal balances, though this aspect would be refined by subsequent Chinchilla scaling laws research.

The methodology involved training models systematically across different scales and measuring performance carefully. The researchers used transformer architectures similar to GPT-2, varying the number of layers, the width of layers, and the size of the training dataset. By controlling for different factors and measuring test loss across all configurations, they could isolate how each factor contributed to performance improvements. This systematic approach enabled them to discover the power-law relationships that governed scaling.

The researchers also investigated how these scaling laws applied to downstream tasks, not just language modeling loss. They found that improvements in language modeling loss translated predictably to improvements on downstream tasks like question answering, reading comprehension, and other NLP benchmarks. This finding suggested that the scaling laws captured fundamental properties of how language models learn, not just properties of the language modeling objective itself. The predictable relationship between pretraining loss and downstream task performance made the scaling laws broadly applicable.

The solution provided concrete mathematical tools for planning and predicting model development. Researchers could now calculate approximately how many parameters and how much training data would be needed to reach a target loss, or predict how much better a model would perform if given additional resources. These predictions enabled more efficient resource allocation and reduced uncertainty in model development planning. The scaling laws transformed model development from relying on intuition and guesswork to using principled mathematical predictions.