Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Causal Language Modeling › Scaling Properties

Michael BrenndoerferUpdated July 7, 20255 min read

Language AI HandbookAll chapters

Part 22 of 67Pre-training Objectives

Chapter 155Causal Language Modeling › Scaling Properties

Reading section 8 of 13 · Scaling Properties · 5 minSection 8 of 13 · 5 min

Section 8 of 13 · 5 min read

Causal Language Modeling › Scaling PropertiesOne section · Sections in this chapter8 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Perhaps the most remarkable property of CLM is how predictably it scales. As we increase model size, dataset size, and compute, performance improves following consistent power laws. This predictability is what distinguishes language model scaling from most engineering disciplines. Usually, when you make something bigger, you encounter bottlenecks, diminishing returns, and phase transitions that are difficult to predict. CLM scaling instead follows remarkably smooth trajectories, making it possible to predict the behavior of a trillion-parameter model by fitting a curve to experiments with much smaller models.

The scaling laws discovered by Kaplan et al. (2020) and refined by Hoffmann et al. (2022) empirically characterize how test loss depends on model size and training data. The key finding is that loss follows a power-law relationship with both factors:

L(N, D) \approx \left(\frac{N_c}{N}\right)^{\alpha_N} + \left(\frac{D_c}{D}\right)^{\alpha_D} + L_\infty

where:

$L(N, D)$ : the cross-entropy loss on held-out test data, as a function of model size and data
$N$ : the number of trainable model parameters (excluding embeddings)
$D$ : the number of training tokens the model has seen
$\alpha_N \approx 0.076$ : the scaling exponent for model size (how quickly loss improves as $N$ grows)
$\alpha_D \approx 0.095$ : the scaling exponent for data (how quickly loss improves as $D$ grows)
$N_c$ and $D_c$ : fitted constants that set the scale (roughly $8.8 \times 10^{13}$ and $5.4 \times 10^{13}$ respectively)
$L_\infty$ : the irreducible loss, representing fundamental uncertainty in language that no model can eliminate

The formula has three additive terms. The first term $\left(\frac{N_c}{N}\right)^{\alpha_N}$ captures model capacity limitations: smaller models have higher loss. The second term $\left(\frac{D_c}{D}\right)^{\alpha_D}$ captures data limitations: less training data means higher loss. The third term $L_\infty$ is the floor, around 1.69 nats, representing the inherent unpredictability of natural language.

This equation reveals that loss decreases as a power law with both model size and data. There's no plateau in sight: 10x more compute yields roughly 0.1 lower loss, consistently across many orders of magnitude.

Out[23]:

Visualization

Log-log plot showing power law decrease in loss as compute budget increases from 10^18 to 10^24 FLOPs. — Theoretical scaling behavior of language model loss with compute budget. The y-axis shows cross-entropy loss (lower is better), while the x-axis shows compute in FLOPs on a log scale. Following Chinchilla scaling, larger compute budgets should be split roughly equally between model size and training tokens.

The practical implication is clear: if you want a better language model, train a bigger model on more data with more compute. This insight has driven the race to scale, from GPT-2's 1.5 billion parameters to models with hundreds of billions.

The Chinchilla Finding and Compute-Optimal TrainingLink Copied

The original Kaplan et al. scaling laws suggested that given a fixed compute budget, you should spend most of it on model size and train on relatively little data. GPT-3, trained with 175 billion parameters on roughly 300 billion tokens, reflected this philosophy.

Hoffmann et al. (2022), in a paper introducing the Chinchilla model, revised this picture significantly. Their experiments showed that the optimal training strategy is to scale model size and data size proportionally. For every doubling of parameters, you should also double the number of training tokens. By this analysis, GPT-3 was substantially undertrained: a much smaller model trained on far more data would have achieved equal performance at much lower cost.

This finding reoriented the field. Subsequent models like LLaMA and Mistral, along with Gemma, were designed with the Chinchilla recipe in mind, training smaller models on larger datasets. A 7-billion-parameter model trained on 2 trillion tokens can outperform a 175-billion-parameter model trained on 300 billion tokens at a fraction of the inference cost. The same compute budget can produce a much more efficient model if allocated correctly between parameters and data.

In practice, the Chinchilla optimal point assumes you train the model once and then stop. But inference matters too. If a model will be deployed and run a billion times, a smaller model is much more valuable than a larger one with equal loss, because every inference query is cheaper. This "inference-optimal" perspective has pushed the field even further toward smaller, data-rich models.

Scaling also improves emergent capabilities. Models below a certain size cannot perform multi-step reasoning, follow complex instructions, or write working code. Above threshold scales, these abilities appear suddenly, a phenomenon called emergence. Whether emergence is truly discontinuous or an artifact of how we measure it remains debated, but the practical reality is clear: beyond certain parameter and data thresholds, qualitatively new behaviors appear that could not be reliably predicted from smaller-scale experiments.