Kaplan Scaling Laws › Loss vs. Compute

The third scaling law relates loss directly to the compute budget. This is perhaps the most practically important relationship because compute (measured in GPU-hours, dollars, or energy consumption) is the fundamental constraint that organizations face when training models. Unlike parameters, which can be chosen freely, or data, which can often be collected or synthesized, compute represents a hard resource limit.

Understanding how loss scales with compute means recognizing that compute is a derived resource, not a fundamental one. You don't directly spend compute; instead, you spend it indirectly by choosing a model size (which determines compute per forward-backward pass) and a number of training steps (which determines how many passes you perform). The compute budget constrains the product of these choices, creating a tradeoff. You can train a large model for few steps or a small model for many steps, but you cannot train a large model for many steps within a fixed compute budget.

When training is optimally allocated between model size and training steps, the loss follows:

$$L(C) = \left(\frac{C_c}{C}\right)^{\alpha_C}$$

where:

• $L$: the cross-entropy loss on held-out text
• $C$: the total compute budget measured in PetaFLOP-days (one PetaFLOP-day equals $10^{15}$ floating-point operations performed continuously for one day)
• $C_c$: a fitted constant representing the characteristic compute scale (approximately $3.1 \times 10^8$ PetaFLOP-days)
• $\alpha_C$: the compute scaling exponent (approximately 0.050)

The characteristic compute scale $C_c \approx 3.1 \times 10^8$ PetaFLOP-days represents an enormous amount of computation, roughly 310 million PetaFLOP-days. To put this in perspective, training GPT-3 required approximately 3,640 PetaFLOP-days, which is about five orders of magnitude smaller than $C_c$. This large characteristic scale ensures that practical training runs operate in the regime where the ratio $C_c/C$ is much greater than 1, producing meaningful loss values that decrease predictably with more compute.

The compute exponent is smaller than both $\alpha_N$ and $\alpha_D$ because compute is a derived resource. It gets split between making the model larger (more parameters) and training longer (more data passes). The compound effect means each 10x increase in compute yields less loss reduction than a 10x increase in parameters or data alone.

The fitted parameters are:

• $\alpha_C \approx 0.050$
• $C_c \approx 3.1 \times 10^8$

To understand why the compute scaling works, consider that total compute is approximately $C \approx 6ND$. The factor of 6 accounts for roughly two operations per parameter in the forward pass and four in the backward pass (computing gradients for both activations and weights). This relationship comes from the arithmetic of matrix multiplications: each parameter participates in operations during both forward and backward passes, with the backward pass requiring roughly twice the computation of the forward pass to compute gradients. The factor of 6 is an approximation that works reasonably well across typical transformer architectures and training configurations, though the exact multiplier can vary with specific implementation details. This means compute constrains the product of model size and data, forcing a tradeoff between the two.

The compute exponent $\alpha_C = 0.050$ is smaller than both $\alpha_N$ and $\alpha_D$. This makes sense: compute is spent on both larger models and more training steps, so its effect is spread across multiple factors. When you increase compute, you gain both from having more parameters (benefiting from the $\alpha_N$ exponent) and from seeing more data (benefiting from the $\alpha_D$ exponent), but neither benefit is realized fully because the compute is divided between them. The key insight is that this relationship only holds when you allocate compute optimally, which brings us to Kaplan's most consequential finding.