Kaplan Scaling Laws › The Experimental Foundation
Language AI HandbookAll chapters Kaplan Scaling Laws › The Experimental FoundationOne section · Sections in this chapter2 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Before examining the specific scaling laws, it helps to understand how these relationships were discovered. The Kaplan team trained hundreds of transformer language models across a wide range of sizes, from 768 parameters to 1.5 billion parameters. They varied three key quantities systematically:
- Model parameters (N): The total number of trainable weights in the network, excluding embedding parameters
- Dataset size (D): The number of tokens in the training corpus
- Compute budget (C): The total floating-point operations used for training, measured in PetaFLOP-days
All models used the same architecture (decoder-only transformers), the same dataset (WebText2), and the same training procedure. This experimental design let them isolate each variable's effect and find clean power-law relationships.
Kaplan et al. counted only non-embedding parameters when measuring model size. They excluded both input embeddings and output embeddings (weight tying was used) because embedding parameters scale differently with vocabulary size. When comparing to their equations, ensure you're using the same counting convention.