Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Kaplan Scaling Laws › Worked Example: Allocating a Compute Budget

Michael BrenndoerferJanuary 13, 20253 min read

Language AI HandbookAll chapters

Part 23 of 67Scaling Laws

Chapter 163Kaplan Scaling Laws › Worked Example: Allocating a Compute Budget

Reading section 8 of 15 · Worked Example: Allocating a Compute Budget · 3 minSection 8 of 15 · 3 min

Section 8 of 15 · 3 min read

Kaplan Scaling Laws › Worked Example: Allocating a Compute BudgetOne section · Sections in this chapter8 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's work through a concrete example to see how these scaling laws guide practical decisions. Suppose you have 10,000 PetaFLOP-days of compute available, a substantial budget representing perhaps several million dollars worth of GPU time. How should you allocate it between model size and training data?

First, establish a baseline calibration. The scaling laws provide proportionalities rather than absolute values, so we need reference points to anchor our calculations. From Kaplan's experiments, a reasonable reference is that 100 PetaFLOP-days is roughly appropriate for training a 10 million parameter model on 1 billion tokens. These numbers serve as our calibration baseline. We'll scale up from this known point using the power-law exponents.

First, we need baseline calibration points. From Kaplan's experiments, a reasonable reference is that 100 PetaFLOP-days is roughly appropriate for training a 10 million parameter model on 1 billion tokens. Using the scaling exponents:

N_{opt}(C) = N_{base} \times \left(\frac{C}{C_{base}}\right)^{0.73}

D_{opt}(C) = D_{base} \times \left(\frac{C}{C_{base}}\right)^{0.27}

where:

$N_{opt}(C)$ : the optimal parameter count for compute budget $C$
$D_{opt}(C)$ : the optimal token count for compute budget $C$
$N_{base}, D_{base}$ : reference values at a known compute level (here, 10M parameters and 1B tokens)
$C_{base}$ : the reference compute level (here, 100 PetaFLOP-days)
The ratio $C/C_{base}$ : represents how many times larger your budget is compared to the reference
The exponents 0.73 and 0.27: the Kaplan optimal allocation exponents, determining how aggressively each resource scales with compute

The calculation proceeds by first computing the ratio of our budget to the baseline budget: $C/C_{base} = 10000/100 = 100$ . This tells us we have 100 times more compute than the baseline case. We then raise this ratio to each power-law exponent to determine how much each resource should scale. For parameters, $100^{0.73} \approx 28.8$ , meaning we should use approximately 29 times more parameters than the baseline. For data, $100^{0.27} \approx 3.5$ , meaning we should use approximately 3.5 times more data than the baseline.

In[13]:

Code

# Given compute budget
C = 10000  # PetaFLOP-days

# Baseline calibration
C_base = 100  # PF-days
N_base = 1e7  # 10M parameters
D_base = 1e9  # 1B tokens

# Calculate optimal allocation using Kaplan exponents
N_optimal = N_base * (C / C_base) ** 0.73
D_optimal = D_base * (C / C_base) ** 0.27

Out[14]:

Console

Compute budget: 10,000 PetaFLOP-days

Kaplan optimal allocation:
  Model size: 0.29 billion parameters
  Dataset: 3.5 billion tokens
  Tokens per parameter: 12.0

For comparison, scaling factors from baseline:
  Model scale: 28.8x larger than 10M baseline
  Data scale: 3.5x larger than 1B baseline

With 10,000 PetaFLOP-days, Kaplan's scaling laws recommend training roughly a 1.4 billion parameter model on about 4.3 billion tokens. The tokens-per-parameter ratio is particularly telling: at only about 3 tokens per parameter, the model sees each parameter's worth of data only a few times. This low ratio reflects the "train big, don't overtrain" philosophy that emerged from this work. Traditional machine learning wisdom suggested that models need many samples per parameter to avoid overfitting, but Kaplan's analysis showed that for language models, the opposite approach works better. Make the model large and stop training relatively early.

Notice the tokens-per-parameter ratio is only about 3. The model sees each parameter's worth of data only a few times. This low ratio reflects the "train big, don't overtrain" philosophy that emerged from this work.