Data-Constrained Scaling › Optimal Repetition Strategies

Since repetition has diminishing returns, how should practitioners allocate resources when data is limited? The answer depends on available compute, dataset size, and target model size.

Consider a fixed compute budget $C$ and dataset size $D$. You can train a smaller model for more epochs or a larger model for fewer epochs. The optimal choice balances three factors:

Model capacity determines how much information can be extracted per example. Larger models learn more from each pass.

Epoch diminishing returns means additional passes yield less. The penalty is steeper for larger models.

Compute cost scales with both model size and tokens processed. Larger models cost more per token.

The optimal strategy is to train the largest model that can complete a "reasonable" number of epochs within your compute budget. "Reasonable" depends on the repetition penalty: Muennighoff et al. suggest that performance degrades unacceptably beyond approximately 4 epochs for large models.

Based on empirical studies, practitioners should consider several guidelines when operating under data constraints:

• When unique data is sufficient for compute-optimal training, use it without repetition. This is the ideal scenario where Chinchilla ratios apply directly.

• When moderate data limitation exists (requiring 2-4 epochs), proceed with training but expect approximately 10-20% higher loss than the compute-optimal equivalent with unlimited data.

• When severe data limitation exists (requiring 8+ epochs), consider these alternatives before extensive repetition: reduce model size to require fewer tokens, invest compute in data quality improvement and deduplication, or explore synthetic data augmentation.

An underappreciated finding from scaling research is that data quality and quantity are not simply additive. Smaller amounts of high-quality data often outperform larger amounts of lower-quality data, and this advantage compounds with repetition.

Consider a choice between 50 billion tokens of carefully curated text (books, Wikipedia, academic papers) and 200 billion tokens of raw web crawl. Training on the curated data with 4 repetitions (200 billion total tokens) typically outperforms single-epoch training on raw crawl, despite the latter having four times the unique examples. The curated data benefits from higher information density per token and cleaner signal for learning linguistic patterns.

This suggests that when facing data constraints, investment in data curation may yield better returns than acquiring more raw data.