Chinchilla Scaling Laws › Chinchilla vs Kaplan: Understanding the Discrepancy

The Chinchilla and Kaplan scaling laws arrive at different conclusions despite both being grounded in extensive experiments. Understanding why helps clarify the details of scaling analysis. This is not a case of one paper being right and the other wrong in some absolute sense. Both papers conducted large-scale experiments and fit power laws to their data. The discrepancy arises from differences in scope, methodology, and the sensitivity of power-law fitting to choices that might seem inconsequential but turn out to matter quite a lot at large scales.

Think of fitting a scaling law as like estimating the slope of a road by measuring its rise and run over a short stretch. If you measure over 100 meters, you get an estimate. If you later measure over 10 kilometers, you might find the slope is slightly different because the road curves. Neither measurement is wrong. They describe the road in different local regions. Power-law exponents for neural network scaling are similar: they are estimated from a finite range of experiments, and extrapolating beyond that range introduces uncertainty. The Kaplan and Chinchilla papers both estimated these exponents, but from different ranges and under different experimental conditions, and the resulting exponents diverged in ways that matter enormously at the scales where frontier models are trained.

The discrepancy had enormous practical consequences. Researchers and organizations that followed Kaplan's guidance trained models like GPT-3 (175B parameters, 300B tokens) and Gopher (280B parameters, 300B tokens) with parameter-to-token ratios far below what Chinchilla would recommend. The resources spent training those models were not wasted, since the models performed well, but they were not as efficiently spent as they could have been. Understanding the source of the discrepancy is therefore not merely academic: it speaks directly to how future scaling research should be designed and interpreted.

The Kaplan experiments focused primarily on smaller models (up to a few hundred million parameters) with relatively limited compute budgets. The Chinchilla experiments covered a broader range, including models up to 16 billion parameters trained with significantly more compute.

Power law exponents estimated from one regime may not extrapolate to others. The Chinchilla team's larger-scale experiments revealed behavior that wasn't apparent in the smaller-scale Kaplan analysis.

The two studies used different training configurations:

The Kaplan analysis suggested that large models can be "early stopped" effectively, meaning you don't need to train them to convergence to achieve good performance. However, this early stopping analysis appears to have underestimated the benefits of longer training.

One key difference involves learning rate scheduling. The Kaplan experiments often trained models for fewer steps than optimal, and learning rate schedules were not always tuned for the full training duration. The Chinchilla experiments used carefully tuned cosine learning rate schedules that decayed properly over the intended training length.

When you tune your learning rate schedule specifically for longer training runs, the benefit of additional training data increases. Models can extract more signal from the data because the optimization process is better matched to the training duration.

Both analyses fit loss as a function of parameters and data, but with different assumptions about the functional form. The Kaplan decomposition allowed for interaction terms and considered how loss varies in suboptimal regimes differently than Chinchilla.

When fitting power laws to noisy data, small differences in the fitting procedure can lead to different exponent estimates, especially when extrapolating beyond the observed range.