GPT-3 › The Chinchilla Insight: Was GPT-3 Undertrained?

GPT-3 was released in 2020, and at the time, training a 175B parameter model on 300 billion tokens seemed extraordinarily ambitious. But within two years, research showed that this configuration was suboptimal from a compute-efficiency standpoint. The Chinchilla paper (Hoffmann et al., 2022) presented a systematic study of scaling laws with more careful experimental controls than previous work, and arrived at a striking conclusion: for a given compute budget, you should train a much smaller model on much more data than GPT-3 used.

The key finding was that GPT-3's 175B parameters were paired with only 300 billion training tokens, roughly 1.7 tokens per parameter. The Chinchilla scaling laws suggested that optimal compute-efficient training requires approximately 20 tokens per parameter. Under this prescription, training a 175B parameter model optimally would require about 3.5 trillion tokens, over ten times what GPT-3 used. Equivalently, for the same compute budget that GPT-3 consumed, a model with around 70B parameters trained on 1.4 trillion tokens would achieve better performance than GPT-3 on downstream tasks.

This result had enormous practical implications. It suggested that the scaling race driven by GPT-3's success was partially misguided: organizations racing to train ever-larger models were getting worse performance per FLOP than they could achieve with smaller, better-trained models. The subsequent generation of models, including Meta's LLaMA family and Google's Gemma family, were designed around Chinchilla-optimal or near-optimal training configurations, training smaller models for much longer on more data.

For GPT-3's in-context learning capability specifically, the Chinchilla finding raises an interesting question: would a Chinchilla-optimal model of the same compute budget show stronger or weaker in-context learning? The few-shot learning capability observed in GPT-3 might partly be a function of the model's size rather than its training quality. A smaller model trained on more data might have better world knowledge and more reliable language understanding, but might require a larger model size before in-context learning emerges. The relationship between model size, training data quality, and in-context learning capability is not fully resolved and remains an active area of research.

What the Chinchilla work does establish is that the scaling laws formalized with the help of GPT-3's results are practically useful: they enable rational planning of training runs rather than ad-hoc scaling. The ability to predict in advance what level of performance a given compute budget can achieve transformed how large language model training was approached, turning it from a largely empirical process into something more like applied engineering.