Data-Constrained Scaling › Limitations and Practical Implications

The data-constrained scaling framework shows important facts about how language models are developing. Researchers and practitioners need to understand these limitations.

The most important limitation is that no amount of repetition, augmentation, or clever optimization can substitute for diverse data. The scaling laws we've discussed treat tokens as fungible units, but real learning requires exposure to different concepts and contexts, not just repeated phrasing. A model that has seen "the cat sat on the mat" a million times will not learn the same things as a model that has seen a million different sentences once each. This suggests that as we approach the data wall, progress will increasingly depend on finding new data sources (private corpora, multimodal data, or entirely new data creation approaches) rather than on better utilization of existing public text.

Synthetic data offers partial relief but introduces new risks. While carefully curated synthetic data can exceed the value of natural web crawl token for token, it cannot capture the full diversity of human knowledge and expression. Models trained heavily on synthetic data may excel at the patterns their generators were good at producing while developing blind spots in areas where generators struggled. The model collapse phenomenon further suggests that synthetic-heavy training regimes require careful monitoring and mixing with natural data anchors.

Organizations facing data constraints should prioritize several strategies. Data quality filtering typically provides better returns than acquiring more low-quality data. Deduplication, though computationally expensive, effectively increases unique data availability. Investment in data curation (removing noise, improving formatting, and filtering for informational value) can shift the effective data multiplier substantially. When repetition is unavoidable, limiting to 4 to 8 epochs and training the largest model that budget allows tends to optimize the quality-compute tradeoff.

Looking ahead, the data constraint may reshape how the field approaches scaling. We explore in the chapter on inference scaling how some researchers are shifting compute investment from training to inference time, trading pre-training FLOPs for test-time computation. This represents one potential response to the training data ceiling. If we cannot acquire more diverse training signal, perhaps we can extract more capability through longer inference chains.