Kaplan Scaling Laws › Loss vs. Data

The second scaling law describes how performance improves with more training data. This addresses a complementary question to parameter scaling: if you gather more text to train on, how much better will your model become? Understanding this relationship matters because data collection has its own costs. Curating high-quality training corpora requires substantial effort, and some domains have limited data availability.

The intuition behind data scaling is similar to parameter scaling but works differently. When you train on a small dataset, the model sees only a limited sample of language patterns. Common constructions appear often enough to learn well, but rarer phenomena (unusual vocabulary, domain-specific expressions, complex reasoning patterns) may appear too infrequently for reliable learning. As the dataset grows, these rare patterns appear more often, giving the model chances to learn them. Eventually, even very large datasets become saturated. The most common patterns are already learned, and additional data mainly provides more examples of patterns you already captured rather than new information.

When the model is large enough that it won't overfit (effectively infinite capacity), the loss follows:

$$L(D) = \left(\frac{D_c}{D}\right)^{\alpha_D}$$

where:

• $L$: the cross-entropy loss on held-out text
• $D$: the number of tokens in the training set (the total amount of text the model learns from)
• $D_c$: a fitted constant representing the characteristic data scale (approximately $5.4 \times 10^{13}$ tokens)
• $\alpha_D$: the data scaling exponent (approximately 0.095), governing how quickly loss improves with more data

The formula's structure mirrors the parameter scaling law, with $D_c$ serving as the characteristic data scale, similar to how $N_c$ is the characteristic parameter scale. This parallel structure isn't coincidental. It reflects the symmetric role that model capacity and data quantity play in determining what a neural network can learn. The characteristic scale $D_c \approx 5.4 \times 10^{13}$ tokens represents roughly 54 trillion tokens, an enormous amount of text that far exceeds any practical training corpus. This large value ensures that the ratio $D_c/D$ remains well above 1 for realistic datasets. This produces the predictable decay in loss as data increases.

This follows the same intuition as parameter scaling: each additional training token gives diminishing returns, but the larger exponent ($\alpha_D > \alpha_N$) means loss responds more to data increases than parameter increases when measured separately.

The fitted parameters are:

• $\alpha_D \approx 0.095$
• $D_c \approx 5.4 \times 10^{13}$

Notice that $\alpha_D > \alpha_N$ (0.095 vs 0.076). This comparison shows something important about how sensitive loss is to each resource. With a larger exponent, each multiplicative increase in data produces a bigger drop in loss than the same increase in parameters. Specifically, for the same increase (say, 10x), data scaling reduces loss by a factor of $10^{0.095} \approx 1.24$ while parameter scaling reduces it by only $10^{0.076} \approx 1.19$. Loss improves faster per unit increase in data than in parameters when measured separately. However, as we'll see shortly, this doesn't mean you should prefer more data over larger models when compute is constrained, because the cost of processing more data must be factored into the optimization.