Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Kaplan Scaling Laws › Predicting Performance at Scale

Michael BrenndoerferJanuary 13, 20254 min read

Language AI HandbookAll chapters

Part 23 of 67Scaling Laws

Chapter 163Kaplan Scaling Laws › Predicting Performance at Scale

Reading section 10 of 15 · Predicting Performance at Scale · 4 minSection 10 of 15 · 4 min

Section 10 of 15 · 4 min read

Kaplan Scaling Laws › Predicting Performance at ScaleOne section · Sections in this chapter10 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

One of the most useful applications of scaling laws is extrapolation. If the power laws hold, you can predict performance at scales you haven't yet trained. This turns scaling laws from retrospective descriptions into planning tools. Rather than running many expensive experiments to find the best configuration, you can train a few small models, fit the scaling parameters, and extrapolate to predict how larger models will perform.

Kaplan et al. used this approach to estimate the performance of models orders of magnitude larger than any that existed at the time. This extrapolation informed the decision to build GPT-3 and provided confidence that the investment would yield proportional returns.

In[17]:

Code

# Predict loss for hypothetical future models
future_params = np.array([1e9, 10e9, 100e9, 1e12])  # 1B to 1T parameters
predicted_loss = (N_c / future_params) ** alpha_N

# Calculate improvement relative to a 100M parameter baseline
baseline_N = 1e8
baseline_loss = (N_c / baseline_N) ** alpha_N
improvement = 1 - (predicted_loss / baseline_loss)

# Calculate the 1T vs 100M improvement for verification
improvement_1T_vs_100M = 1 - ((N_c / 1e12) ** alpha_N) / (
    (N_c / 1e8) ** alpha_N
)

Out[18]:

Console

Predicted loss vs. model scale (Kaplan laws):
---------------------------------------------
     1B params: Loss = 2.3756 (16.1% improvement over 100M)
    10B params: Loss = 1.9943 (29.5% improvement over 100M)
   100B params: Loss = 1.6741 (40.8% improvement over 100M)
  1000B params: Loss = 1.4053 (50.3% improvement over 100M)

These predictions show how loss decreases as models scale from 1 billion to 1 trillion parameters. The improvement percentages represent how much lower the loss is compared to a 100 million parameter baseline. Each 10x increase in parameters yields approximately a 16% relative improvement, consistent with the $\alpha_N = 0.076$ exponent.

Out[19]:

Visualization

Log-log plot showing predicted loss versus parameters with observed region highlighted. — Extrapolating loss predictions across model scales. The shaded region shows the range observed in Kaplan's original experiments, while the extended line shows predictions for larger models. This extrapolation capability enabled planning for models like GPT-3 before committing resources.

At 1 trillion parameters, the model achieves roughly 16% lower loss than the 100M baseline, a meaningful improvement that reflects the diminishing returns inherent in power-law scaling. While 16% might sound modest, remember that cross-entropy loss relates logarithmically to perplexity. A 16% reduction in loss translates to a substantial reduction in perplexity, which correlates with noticeably better text generation quality. This finding influenced strategic decisions about how far to push scale.

How reliable these extrapolations are depends on whether the power-law relationships continue to hold at larger scales. If there are phase transitions, saturation effects, or architectural limitations that emerge at scale, the predictions could be systematically wrong. This uncertainty motivated ongoing research into whether scaling laws exhibit any departures from pure power-law behavior.