Kaplan Scaling Laws › Limitations and Impact

The Kaplan scaling laws helped us understand neural language models, but they had significant limitations that later research would address.

The main limitation was their experimental methodology for measuring optimal allocation. Kaplan et al. estimated optimal model and data sizes by training models for different numbers of steps and seeing which configurations gave the best loss for a given compute budget. However, they did not train any models fully to convergence. They relied on early training dynamics to extrapolate optimal configurations. This choice led to systematic bias toward larger models and shorter training runs. The Chinchilla scaling laws, which we'll explore in the next chapter, would later demonstrate that training models much longer on more data is more compute-efficient than Kaplan's analysis suggested.

Another limitation was the narrow scope of experiments. All experiments used a single architecture family (decoder-only transformers), a single dataset (WebText2), and a single domain (English web text). The fitted constants and even the exponents may not generalize to encoder models, different languages, or specialized domains like code or scientific text. Subsequent work has shown that optimal scaling behavior does vary across these dimensions.

The treatment of compute was also simplified. Kaplan measured compute in PetaFLOP-days, aggregating all operations equally. In practice, different operations have different hardware efficiency, and memory bandwidth often matters more than raw FLOPS for large models. The relationship $C \approx 6ND$ that Kaplan used as an approximation is reasonable but not exact across all architectures and batch sizes.

Despite these limitations, Kaplan's work had a major impact. For the first time, researchers had quantitative guidance for resource allocation decisions. The framework of power-law scaling became standard vocabulary in the field, and the methodology of fitting scaling laws to predict large-model performance became essential for compute-limited research planning. GPT-3's architecture and training decisions were directly informed by these scaling laws. This shows their practical influence on billion-dollar training runs.

The work also raised questions that motivate ongoing research: Why do power laws emerge at all? Are there phase transitions or inflection points hidden at larger scales? What determines the exponents, and can we design architectures with better scaling? These questions relate to the emergence phenomena we'll explore in later chapters.