Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Kaplan Scaling Laws › Comparing Scaling Behavior

Michael BrenndoerferJanuary 13, 20253 min read

Language AI HandbookAll chapters

Part 23 of 67Scaling Laws

Chapter 163Kaplan Scaling Laws › Comparing Scaling Behavior

Reading section 9 of 15 · Comparing Scaling Behavior · 3 minSection 9 of 15 · 3 min

Section 9 of 15 · 3 min read

Kaplan Scaling Laws › Comparing Scaling BehaviorOne section · Sections in this chapter9 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

To appreciate how the three scaling laws interact and to develop intuition for their relative magnitudes, it helps to visualize them together on a common scale. This comparison shows the hierarchy of exponents and explains why optimal compute allocation favors model size over data.

When we plot all three scaling relationships on the same axes, we normalize each curve to pass through the point (1, 1), representing a baseline case. This allows us to compare slopes: a steeper slope means faster loss improvement per unit increase in the resource. The steepness of each curve directly reflects its corresponding exponent. Larger exponents produce steeper downward slopes on log-log axes.

In[15]:

Code

# Create normalized comparison of all three scaling laws
# Normalize so all curves pass through (1, 1) for comparison

# Relative scaling (normalized to 1 at leftmost point)
x_relative = np.logspace(0, 4, 100)  # 1x to 10000x

# Loss improvement factors using Kaplan exponents
L_N_relative = x_relative ** (-alpha_N)  # Parameters
L_D_relative = x_relative ** (-alpha_D)  # Data
L_C_relative = x_relative ** (-alpha_C)  # Compute

Out[16]:

Visualization

Log-log plot comparing three declining curves for data, parameters, and compute scaling. — Comparison of scaling law exponents showing relative loss improvement from each factor. Data scaling (α=0.095) improves loss fastest per unit increase, followed by parameters (α=0.076), then compute (α=0.050).

At first glance, you might think this figure contradicts the recommendation to scale models over data. After all, the data curve drops fastest. Remember that the constraint is compute, not the resource itself. This is a subtle but important distinction. The x-axis shows multiplicative increases in each resource independently, but in practice, you cannot independently scale data without consuming compute. Increasing parameters by 10x costs proportionally more compute than increasing data by 10x (roughly). When you factor in these costs, the optimal allocation still favors larger models.

The answer is that data scaling requires compute (you must process each token), while parameter scaling has a fixed cost per training step regardless of model size. Roughly speaking, the compute per step scales with $N$ , but the total compute $C \approx 6ND$ means the relationship is multiplicative. The optimal allocation calculation properly accounts for these costs and determines that investing in model size yields better returns per unit of compute than investing in more training data.