Data-Constrained Scaling › Data Augmentation at Scale

Traditional NLP relied heavily on data augmentation, transforming existing examples to create new training instances. Techniques like back-translation, synonym substitution, and random deletion could multiply dataset size by 10× or more. How do these techniques interact with scaling?

The most common augmentation techniques for language modeling include:

• Back-translation: Translates text to another language and back, creating paraphrases that preserve meaning but vary in surface form. A sentence like "The cat sat on the mat" might become "The feline rested upon the rug" after round-trip translation through French.

• Synonym substitution: Replaces words with synonyms using a thesaurus or word embeddings. This creates lexical variation while maintaining semantic content.

• Random operations: Delete or insert tokens, or swap their positions, to add noise while preserving most structure. Deleting 10% of words or swapping adjacent words creates varied examples.

• Paraphrase generation: Uses trained paraphrase models to rewrite sentences with different syntactic structures.

Augmentation provides less value at scale than researchers initially hoped. Several studies found that augmented data behaves more like repeated data than unique data. This makes sense. Augmentations preserve the underlying semantic content while varying surface features. A model quickly learns to see through superficial variations, treating augmented examples as near-duplicates.

To understand this phenomenon more deeply, consider what a language model learns. At a superficial level, it learns associations between word sequences. But at a deeper level, it learns semantic relationships, factual knowledge, and reasoning patterns. Augmentation techniques typically preserve the deeper content while only varying the surface form. A model sophisticated enough to benefit from massive scale is also sophisticated enough to recognize that "The cat sat on the mat" and "The feline rested upon the rug" convey the same information, even if the tokens differ.

The effective repetition factor for augmented data is approximately 0.3-0.5 compared to unique data. More formally, if we denote the augmentation effectiveness as $\eta \approx 0.3-0.5$, then generating $k$ augmented variants yields effective data:

$$D_{\text{eff}}^{\text{aug}} \approx D \cdot (1 + \eta \cdot (k-1))$$

where:

• $D$: the number of original unique tokens
• $k$: the number of variants per example (original + augmentations)
• $\eta$: the augmentation effectiveness factor (how much value each augmented example provides relative to a unique example)

Let's trace through the logic of this formula. We start with $D$ original tokens, which contribute their full value. We then add $k-1$ augmented variants (since $k$ includes the original). Each augmented variant contributes only $\eta$ times the value of a unique example. So our total effective data is the original $D$ plus the discounted value of the augmentations: $D + D \cdot \eta \cdot (k-1) = D \cdot (1 + \eta \cdot (k-1))$.

An augmentation that produces 5 variants of each example ($k=5$) with $\eta=0.4$ might yield effective data equivalent to $D \cdot (1 + 0.4 \cdot 4) = 2.6D$, approximately 2.6 repetitions worth rather than 5 unique examples. This shows a significant gap: you have done five times the computational work of processing unique data but gained less than three times the effective training signal.

This doesn't mean augmentation is useless, but it is not a straightforward improvement over raw repetition. At typical augmentation effectiveness levels (η ≈ 0.3–0.5), repeating the original data yields higher effective token counts than generating augmented variants at the same computational cost. Augmentation's main advantage lies in diversity of surface form, which can help prevent certain failure modes like verbatim memorization, but it does not substitute for unique data. Transforming 100 billion tokens into 1 trillion augmented tokens does not yield performance equivalent to 1 trillion unique tokens.

Augmentation shows the strongest benefits in two scenarios:

• Low-resource settings: Settings where base data is severely limited (less than 1 billion tokens) benefit substantially from augmentation. The model has not fully memorized the training set, so variations provide a learning signal.

• Domain-specific fine-tuning: Scenarios where acquiring more in-domain data is impractical can use augmentation. If you have only 10,000 medical documents, augmentation can help prevent overfitting during fine-tuning.

For large-scale pretraining where data is measured in trillions of tokens, augmentation provides marginal value compared to its computational cost.