Synthetic Data › Summary

Synthetic data generation is the practice of using capable teacher models to produce labeled training examples at scale. It works because generation is easier than learning from scratch, because it provides coverage for underrepresented behaviors, because it delivers high-quality labeled examples at low cost, and because it gives practitioners deliberate control over the training distribution. The key ideas covered in this chapter are:

• Generation methods: Self-Instruct bootstraps from a small seed set using iterative pool expansion; Evol-Instruct evolves instructions to higher complexity through systematic mutations; backtranslation converts high-quality text into instruction-response pairs by generating instructions for existing responses; rejection sampling filters by verifiable correctness, giving a hard quality signal for structured domains
• Quality verification: Rule-based filters catch obvious failures cheaply before any model calls; model-based scoring with a judge model evaluates semantic quality on a numerical scale; IFEval-style format verification checks compliance with explicit constraints programmatically; reward model scoring provides a learned preference signal; consistency filtering uses agreement across multiple generations as evidence for correctness
• Diversity: Measured through embedding-based pairwise distance, vocabulary richness, and topic entropy; topic collapse is the most common failure mode; improved through explicit topic-guided generation, persona-based generation for perspective diversity, and iterative self-play with diversity constraints; near-duplicate removal via MinHash prevents redundant examples from dominating the dataset
• Knowledge distillation: Response distillation trains students on teacher outputs via standard cross-entropy; chain-of-thought distillation transfers reasoning traces rather than just final answers; logit distillation transfers soft probability distributions including dark knowledge; the $T^2$-scaled KL loss combines hard and soft targets; distillation performance degrades when the teacher-student capacity gap is too large, producing style distillation rather than capability distillation

Synthetic data is not a replacement for real data but a complement to it, filling gaps that natural corpora cannot cover and providing targeted training signals for specific capabilities. The most effective training pipelines combine large-scale natural pretraining with carefully curated synthetic fine-tuning data, using each source for what it does best.