Synthetic Data › Limitations and Practical Considerations

Synthetic data is powerful but not without significant failure modes that practitioners need to understand and actively mitigate.

Hallucination propagation is the most serious concern. When a teacher model generates factually incorrect content, that error enters the training data. A student trained on such data learns the error as fact, and the error compounds across training steps as the student's own generations reinforce the incorrect belief. This is especially dangerous for factual knowledge (historical dates, scientific constants, drug dosages, legal facts) where the teacher may be confidently wrong. Hallucinations in a teacher model's output look exactly like correct outputs: they are fluent, confidently stated, and well-integrated into the response. Standard quality filters are nearly blind to them. Mitigation requires grounding verification: checking generated facts against authoritative sources wherever possible, and using rejection sampling with external validators for tasks that have verifiable ground truth.

Capability ceiling limits what distillation can achieve. A student model cannot systematically exceed the teacher's performance on the tasks the synthetic data covers. The ceiling is the teacher's capability level. To raise the ceiling, you need either a better teacher, a different training signal (human feedback, external verification), or tasks where the student can self-improve (mathematical reasoning with verifiable answers, for instance). This ceiling effect has significant practical implications: the widespread availability of GPT-4 as a teacher model has made 7B-70B student models much more capable, but it also means that all students distilled from GPT-4 will plateau at or below GPT-4's capability level.

Distribution mismatch occurs when synthetic data covers the training distribution well but differs from the deployment distribution. If users ask questions in styles, languages, or domains that the synthetic generation pipeline did not anticipate, the trained model may fail. Real user data collected during deployment provides a valuable complement to synthetic data, even in small amounts. A few thousand real user interactions can reveal distribution gaps that no amount of synthetic data generation would have surfaced, because the synthetic data reflects what the pipeline designers anticipated rather than what users do.

Repetition and template fatigue degrade model quality when too high a fraction of training data comes from a single generation pipeline with a single prompt template. The model learns to mimic the template's patterns, producing outputs that feel formulaic. If every training example begins with "Certainly! Here is a detailed explanation of..." and ends with "I hope this explanation was helpful!", the student learns to produce exactly that framing regardless of context. Rotating templates, mixing generation strategies (Self-Instruct alongside Evol-Instruct alongside backtranslation), and blending synthetic data with natural data all mitigate this effect. Analyzing the most common n-grams in your synthetic dataset is a simple diagnostic: if a handful of phrases appear in more than 5% of examples, your templates are leaving too much of a fingerprint.

Evaluation data contamination is a subtle risk: if the same teacher model that generates training data is also used to evaluate the student, the evaluation may be biased. The teacher's responses set the implicit style standard, and the student's outputs may look good simply because they mimic that style, even when they are wrong in ways the teacher-judge cannot detect. Using held-out human evaluators or independent evaluation benchmarks that the teacher model has not seen is essential for unbiased assessment.

Data mixing balance becomes critical when synthetic data constitutes a large fraction of the training mixture. Models trained on 100% synthetic data tend to have peculiar failure modes: they over-represent the teacher's writing style, they perform well on instruction-following benchmarks (which are themselves based on similar distribution assumptions) but poorly on open-ended real-world tasks, and they sometimes exhibit exaggerated confidence without appropriate uncertainty. Blending synthetic data with a substantial fraction of natural pretraining text maintains the breadth and naturalness that large-scale web crawls provide while adding the targeted capability that synthetic data delivers.

Despite these limitations, synthetic data remains one of the most cost-effective tools for improving model capability. The Phi family of models demonstrated that a carefully curated synthetic dataset of a few billion tokens could produce a model competitive with much larger models trained on much larger natural corpora. Phi-1 achieved state-of-the-art performance on Python coding benchmarks with only 1.3B parameters, trained almost entirely on synthetic "textbook quality" data generated by GPT-4. Phi-2, at 2.7B parameters, matched or exceeded models 5-10x its size on reasoning benchmarks, again largely through synthetic training data. As generation quality and verification pipelines improve, synthetic data will continue to grow in importance across the full spectrum of model training.

With the data curation pipeline complete, Part XXXI turns to training infrastructure: the GPU architecture, memory management, and distributed training strategies needed to train large models on the data assembled across Part XX.