Scaling Laws (2020) › Limitations

Despite their significant contributions, the 2020 scaling laws had important limitations that would be addressed by subsequent research. Perhaps most significantly, the laws focused on how performance scaled with individual factors but did not explicitly address the optimal balance between model size and training data for a fixed computational budget. The laws showed that performance improved with both larger models and more training data, but they provided less guidance about how to allocate a fixed computational budget between these two factors. This limitation would be addressed by Chinchilla scaling laws research, which would show that many models were undertrained relative to their size.

The scaling laws were derived from experiments with models up to 1.5 billion parameters, while subsequent models would scale to hundreds of billions of parameters. The question of whether the power-law relationships would hold at much larger scales remained open. While the laws provided reasonable predictions for GPT-3's scale, their applicability to even larger models required validation. Subsequent research would investigate whether scaling laws continued to hold at larger scales or whether new relationships emerged.

The laws also did not address all aspects of scaling. While they captured relationships between loss and model size, dataset size, and compute, they did not explicitly consider factors like model architecture, training procedures, or data quality. The relationships might vary for different architectures or training approaches, and the laws might not capture important nuances in how these factors affect scaling. Understanding these nuances would require additional research beyond the initial scaling laws formulation.

The scaling laws measured performance primarily through cross-entropy loss on language modeling tasks. While the researchers showed that improvements in loss translated to improvements on downstream tasks, the exact relationship might vary across different task types. Some tasks might benefit more from certain aspects of scaling than others, and the laws might not capture all the nuances of how scaling affects different capabilities. Understanding these variations would be important for applying the laws effectively.

The laws also did not address questions about the limits of scaling. While they showed smooth power-law improvements, they did not predict where these relationships might break down. Would the laws continue to hold as models approached the limits of available training data? Would computational constraints eventually limit scaling? Would there be fundamental limits to how much performance could improve through scaling alone? These questions remained open and would drive continued research.

The computational cost implications of the scaling laws raised concerns about accessibility and sustainability. If performance continued to improve with scale following power laws, achieving state-of-the-art performance would require increasingly enormous computational resources. This requirement could concentrate capabilities in the hands of organizations with massive computational budgets, limiting diversity in research directions and potentially creating barriers to entry for smaller organizations or researchers. These concerns would motivate research into more efficient architectures and training methods.

The scaling laws also did not address important aspects of model quality beyond raw performance metrics. The loss metrics did not capture bias or whether a model behaved safely and reliably under operator control. A model that achieved lower loss might not necessarily be more useful or safer in practice. Understanding how these qualitative aspects of model behavior scale would require additional research beyond the initial scaling laws formulation.

The relationships discovered were statistical rather than deterministic. While the laws provided good predictions on average, individual training runs might deviate from these predictions due to factors like random initialization, data ordering, or other sources of variance. The laws were useful for planning and estimation but did not guarantee specific outcomes for individual training runs. This variability meant that the laws were best used as guides rather than precise guarantees.