Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Kaplan Scaling Laws › Optimal Compute Allocation

Michael BrenndoerferJanuary 13, 20256 min read

Language AI HandbookAll chapters

Part 23 of 67Scaling Laws

Chapter 163Kaplan Scaling Laws › Optimal Compute Allocation

Reading section 7 of 15 · Optimal Compute Allocation · 6 minSection 7 of 15 · 6 min

Section 7 of 15 · 6 min read

Kaplan Scaling Laws › Optimal Compute AllocationOne section · Sections in this chapter7 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Kaplan's most influential finding was their guidance on allocating a fixed compute budget between model size and training data. This question is very practical: given a budget of one million dollars worth of GPU time, should you train a modest model for a long time on lots of data, or should you train a very large model for a shorter time on less data? Before scaling laws, practitioners relied on intuition and trial-and-error. Kaplan's analysis provided a principled answer backed by empirical evidence.

The derivation of optimal allocation starts from the fact that compute constrains the product of model size and data: $C \approx 6ND$ . This means that for a fixed compute budget, increasing $N$ necessarily decreases $D$ (the number of training tokens seen). The question becomes: what ratio of $N$ to $D$ minimizes loss for a given $C$ ?

Given a compute budget $C$ , the optimal model size and dataset size follow:

N_{opt} \propto C^{0.73}

D_{opt} \propto C^{0.27}

where:

$N_{opt}$ : the optimal number of parameters for a given compute budget
$D_{opt}$ : the optimal number of training tokens for that same budget
$C$ : the total compute budget in PetaFLOP-days
$\propto$ : denotes proportionality. The left side scales with the right side up to a constant factor
The exponent 0.73: derived from fitting optimal configurations, indicates parameters should grow faster than the square root of compute
The exponent 0.27: the complementary exponent for data scaling, since $0.73 + 0.27 = 1.0$ reflects the constraint $C \propto ND$

These exponents sum to 1.0, which follows from the compute relationship $C \approx 6ND$ . Taking logarithms: $\log C = \log 6 + \log N + \log D$ . If we write $N \propto C^a$ and $D \propto C^b$ , then $\log C = \text{const} + a\log C + b\log C$ , requiring $a + b = 1$ . This mathematical constraint ensures consistency: you cannot independently choose how both parameters and data scale with compute, because compute is defined as their product. Notice the stark imbalance: 73% of the compute scaling goes toward larger models, while only 27% goes toward more data.

The asymmetry in these exponents reflects Kaplan's central finding: larger models are more sample-efficient. A bigger model extracts more learning from each training token it sees. This means that as you scale up compute, you should primarily invest in model size rather than training duration. The intuition is that a larger model has more capacity to represent complex patterns, so it can learn those patterns from fewer examples. A smaller model, by contrast, needs to see the same patterns many times before it can reliably capture them.

Kaplan's Central Recommendation

When your compute budget increases by 10x, you should increase model size by approximately $10^{0.73} \approx 5.4\times$ and training tokens by only $10^{0.27} \approx 1.9\times$ . Larger models are more sample-efficient. They extract more learning from each token they see.

This recommendation drove concrete decisions. For GPT-3, OpenAI chose a 175B parameter model trained on 300B tokens. Following Kaplan's guidance meant prioritizing a very large model over training on more data.

In[10]:

Code

# Kaplan optimal allocation exponents
exp_N = 0.73
exp_D = 0.27

# Calculate optimal allocation across compute budgets
compute_budgets = np.logspace(2, 6, 50)  # 100 to 1M PetaFLOP-days

# Normalize to reasonable scales (these are proportional, not absolute)
# Using rough calibration points from the paper
N_opt = (
    1e7 * (compute_budgets / 100) ** exp_N
)  # Baseline: 10M params at 100 PF-days
D_opt = (
    1e9 * (compute_budgets / 100) ** exp_D
)  # Baseline: 1B tokens at 100 PF-days

Out[11]:

Visualization

Log-log plot showing optimal model parameters versus compute. — Optimal model size scaling with compute budget. Parameters grow as C^0.73, meaning most additional compute should be invested in larger models.

Log-log plot showing optimal dataset tokens versus compute. — Optimal dataset size scaling with compute budget. Tokens grow as C^0.27, indicating relatively modest increases in training data.

The visual contrast is striking. As compute increases 10,000x (from $10^2$ to $10^6$ PF-days), optimal model size increases by roughly 2500x while optimal dataset size increases by only about 20x. This asymmetry reflects Kaplan's core finding: bigger models learn more efficiently.

Out[12]:

Visualization

Line plot showing predicted loss versus fraction of compute allocated to parameters. — Impact of allocation strategy on predicted loss. For a fixed compute budget of 10,000 PetaFLOP-days, different splits between model size and training data yield different losses. The Kaplan-optimal allocation (73% to parameters, 27% to data) achieves the lowest loss, validating the 'scale models over data' recommendation.