Data-Constrained Scaling › Summary

This chapter examined what happens when language model training collides with finite data availability. The key insights are:

Repeated data has diminishing returns. The effective value of the $n$-th pass through data follows an exponential decay, with approximately 60-70% of value extracted in the first two epochs and minimal gains beyond 8-16 epochs.

Larger models suffer more from repetition. Their greater capacity means they extract information faster, leaving less to learn on subsequent passes. This shifts optimal strategies toward fewer epochs with larger models when data is limited.

Data augmentation provides limited relief. Augmented examples behave more like repeated data than unique data, capturing perhaps 30-50% of unique data value.

Synthetic data offers promise with caveats. High-quality synthetic data can exceed web crawl value per token, but requires careful curation and risks model collapse over generations.

Quality dominates quantity at the margin. When data is constrained, investment in curation and filtering often yields better returns than acquiring more raw data.

As training runs grow larger, the data wall will increasingly shape model development strategy. The next chapter on inference scaling explores one response: shifting compute investment from training time to inference time, potentially working around training data limits through more sophisticated test-time computation.