Data-Constrained Scaling › Synthetic Data Scaling

If augmented data provides limited value, what about entirely synthetic data? Modern language models can generate coherent, novel text. Could we use model-generated data to train the next generation of models?

This question has generated intense research interest, with results that are simultaneously promising and concerning.

The appeal is clear. Language models can generate unlimited text. If synthetic data were as valuable as natural data, the data wall would dissolve. Training would be constrained only by compute, and we could generate as much data as needed.

The outcome depends on the generator model, the generation strategy, and how synthetic data is mixed with natural data.

The most successful synthetic data approaches emphasize quality over quantity. Microsoft's Phi-1 model demonstrated this dramatically. A 1.3 billion parameter model trained primarily on synthetic "textbook-quality" data matched or exceeded models ten times larger trained on web crawl.

The key insight was using GPT-4 to generate educational content designed for learning, rather than arbitrary text. The synthetic data consisted of:

• Textbook chapters explaining programming concepts step-by-step
• Exercises with varying difficulty levels
• Synthetic code with clear, well-documented examples

This curated synthetic data proved more valuable token-for-token than natural web data. One token of synthetic textbook was worth approximately ten tokens of random web crawl for learning coding abilities.

How synthetic data is generated matters enormously. Simply prompting a model to "generate text about X" yields bland, repetitive content. Effective strategies include:

• Curriculum-aware generation: Produces data spanning difficulty levels. Easy examples establish foundations; hard examples push capabilities.

• Diverse seed prompting: Uses varied prompts to cover different topics and styles in multiple formats. A thousand different prompt templates yield more diverse output than one template run a thousand times.

• Self-instruct: Has models generate instructions and completions, creating task-diverse training data. This approach powers much of modern instruction tuning.

• Verification filtering: Generates many candidates and filters to high-quality examples. For code, this might mean keeping only examples that compile and pass tests. For math, keeping only solutions that reach correct answers.

Synthetic data also carries risks. Shumailov et al. (2023) identified "model collapse," a phenomenon where models trained on synthetic data from previous model generations progressively degrade.

The mechanism is subtle but important. Each model generation slightly shifts the data distribution toward higher-probability outputs and away from the tails. Rare but important patterns (unusual words, unconventional syntax, domain-specific terminology) become rarer in synthetic data than in natural data.

When model $G_1$ generates training data for model $G_2$, the training distribution shifts. When $G_2$ generates data for $G_3$, the shift compounds. After several generations, the distribution has collapsed toward a narrow subset of high-probability outputs, and models lose capability on anything outside this narrow core.

The implications are concerning. If the internet becomes dominated by AI-generated content, future models trained on web crawls might suffer progressive capability loss.

Several strategies help prevent model collapse:

• Mixing synthetic with natural data: Maintains anchoring to the true distribution. Successful approaches typically use 50-80% natural data even when synthetic data is available.

• Freshness filtering: Preferentially uses synthetic data that is novel relative to the training set. If a generated example is too similar to existing data, discard it.

• Diversity objectives: Explicitly optimize for varied outputs during generation, using techniques like diverse beam search or temperature annealing.

• Watermarking: Tags AI-generated content, allowing future training pipelines to identify and appropriately weight synthetic data.