Data-Constrained Scaling › Data-Constrained Scaling

The Chinchilla scaling laws from our previous chapter offer a straightforward recipe: to train a compute-optimal model, scale parameters and training tokens in roughly equal proportion. A 70 billion parameter model should see approximately 1.4 trillion tokens. But this prescription contains a hidden assumption that becomes increasingly problematic as models grow larger: it assumes we have unlimited unique training data.

In practice, we don't. The internet, despite its vastness, contains a finite amount of high-quality text. As the AI community trains ever-larger models, we're approaching what researchers call the "data wall": the point where compute-optimal training would require more unique tokens than exist. This chapter examines what happens when data becomes the binding constraint, how repeated data affects model quality, and strategies for stretching limited data resources.