Kaplan Scaling Laws › Loss vs. Parameters

The first major finding concerns how test loss decreases as models grow larger. This relationship answers a basic question practitioners face: if I make my model bigger, how much better will it perform? Before scaling laws, researchers had only rough intuitions. More parameters generally helped, but by how much? And for how long?

To understand why a power law might govern this relationship, consider what happens as you add parameters to a neural network. Early parameters learn common language patterns: basic grammar, frequent word associations, and simple semantic relationships. These patterns appear consistently in training data, so the model learns them reliably and they help reduce loss a lot. As you add more parameters, the model learns increasingly subtle and rare phenomena: unusual grammatical constructions, domain-specific terminology, and subtle contextual dependencies. Each new layer of complexity appears less frequently in the data and helps less with performance. This diminishing return is exactly what power laws describe.

When training models to convergence with effectively unlimited data, the relationship follows a power law:

where:

• $L$: the cross-entropy loss on held-out text, measuring how well the model predicts the next token
• $N$: the number of non-embedding parameters (trainable weights excluding embeddings)
• $N_c$: a fitted constant representing the "characteristic scale" of parameters (approximately $8.8 \times 10^{13}$)
• $\alpha_N$: the scaling exponent (approximately 0.076), which determines how quickly loss decreases with more parameters
• $\left(\frac{N_c}{N}\right)$: the ratio of the characteristic scale to the actual model size. When $N$ is small compared to $N_c$, this ratio is large, yielding higher loss

Let's examine this formula's structure. The characteristic scale $N_c$ is a reference point, an astronomically large number of parameters (88 trillion) that is a normalizing constant. Think of $N_c$ as the scale where the ratio $N_c/N$ equals 1. This makes the loss contribution from this term equal to 1. Since no practical model approaches this scale, the ratio $N_c/N$ is always much greater than 1 for real models. This produces loss values that decrease predictably as $N$ grows.

The power-law form appears because each additional parameter gives diminishing returns. The ratio $N_c/N$ captures how far the model is from some theoretical limit, and raising it to a small power $\alpha_N$ yields the smooth, predictable decay we observe. Kaplan et al. found:

• $\alpha_N \approx 0.076$
• $N_c \approx 8.8 \times 10^{13}$

The exponent $\alpha_N = 0.076$ shows exactly how much improvement you get from scale. This small value tells us that each 10x increase in parameters yields a factor of $10^{0.076} \approx 1.19$ reduction in the loss ratio $(N_c/N)^{\alpha_N}$, or equivalently about a 16% decrease in the excess loss. In simpler terms: doubling the model size reduces loss by about 5%. This might seem modest, but loss improvements compound multiplicatively and cross-entropy loss relates exponentially to perplexity. Small loss reductions can mean noticeably better text generation.

The linearity on log-log axes confirms the power-law relationship. This was not obvious beforehand. The loss could have plateaued, shown diminishing returns, or shown more complex behavior. Instead, the relationship holds remarkably cleanly across four orders of magnitude in model size. The straight line you observe when plotting both axes on logarithmic scales is the defining signature of a power law, and its consistency across such a wide range suggests that the underlying phenomenon reflects something fundamental about how neural networks learn from data.