Kaplan Scaling Laws › The Unified Scaling Law

The individual scaling laws describe performance in idealized settings: infinite data, infinite compute, or optimal allocation. In practice, models train with finite resources across all dimensions. A model might have limited parameters due to hardware constraints, limited data due to collection challenges, and limited compute due to budget restrictions. The unified scaling law addresses this realistic scenario by combining all three factors into a single predictive equation.

The challenge in formulating such a law is figuring out how different constraints interact. If you have a large model but limited data, the model may overfit or fail to use its full capacity. If you have abundant data but a small model, the model may underfit and fail to capture the complexity in the data. The unified law must capture these interactions while reducing to the individual laws in the appropriate limits.

Kaplan et al. proposed a unified equation that combines all three factors:

where:

• $L(N, D)$: the predicted loss as a function of both model size and data size
• $N$: non-embedding parameters
• $D$: dataset size in tokens
• $N_c, D_c$: the characteristic scales from the individual scaling laws
• $\alpha_N, \alpha_D$: the scaling exponents (0.076 and 0.095 respectively)
• $\frac{\alpha_N}{\alpha_D}$: the ratio of exponents (approximately 0.8), which balances the relative contributions of model and data limitations
• The brackets $[\cdot]^{\alpha_D}$: the outer exponent applied after summing the two limitation terms, converting the combined constraint into a loss prediction

The formula's structure reveals how parameter and data constraints combine mathematically. Consider the expression inside the brackets: it sums two terms, each representing a different kind of limitation. The term $(N_c/N)^{\alpha_N/\alpha_D} \approx (N_c/N)^{0.8}$ captures the model's capacity limitation, representing how far the current model size is from the characteristic scale, adjusted by the ratio of exponents to ensure consistent units when adding to the data term. The second term, $D_c/D$, captures the data limitation in a parallel fashion. Adding them inside the brackets before applying $\alpha_D$ means that whichever constraint is more binding will dominate. You cannot compensate for too little data with a larger model, or vice versa.

This additive structure inside the brackets creates what mathematicians call a "soft minimum" behavior. When one term is much larger than the other, the sum is dominated by that term, and the loss depends primarily on the binding constraint. When both terms are comparable, both constraints matter, and the loss reflects an interaction between model capacity and data availability. This matches our intuition about how training should behave: you need both sufficient model capacity and sufficient data to achieve good performance.

This formula captures an important insight: there's an effective data-to-model ratio that determines performance. A large model on too little data leads to overfitting (the model can't learn effectively from limited signal). Training a small model on too much data leads to a different kind of inefficiency: the model saturates and additional data provides diminishing returns.

The unified law also reveals the irreducible loss component. Even with infinite parameters and infinite data, some entropy remains in natural language that no model can predict. This represents the fundamental unpredictability of language itself.