Synthetic Data › Why Synthetic Data Works

Before diving into mechanics, it is worth understanding why using model-generated text to train models is not circular. The intuition might seem suspicious: if a model generates the training data, are you not just teaching the model to imitate itself?

The key insight is that generation is easier than learning from scratch. A capable teacher model already contains knowledge and reasoning capability accumulated during its own pretraining. Generating synthetic data is a process of eliciting and rendering that knowledge into structured, labeled examples that a student model can learn from efficiently. The student does not need to rediscover reasoning patterns from unstructured text; it gets curated demonstrations of those patterns, with the structure and framing optimized for learning.

Think of it this way: if you want to teach someone to write formal proofs, you could hand them a thousand pages of raw mathematical literature and let them figure it out, or you could show them five hundred carefully constructed proof examples that isolate exactly the reasoning moves they need to internalize. Both contain roughly the same underlying knowledge, but the second form is dramatically more sample-efficient because it delivers that knowledge at the right level of abstraction for the learner. Synthetic data is the second approach applied at scale.

A second reason synthetic data works is coverage. Natural corpora underrepresent many desirable behaviors: careful step-by-step reasoning, refusals for harmful requests, low-resource language responses, domain-specific technical writing, and structured output formats like JSON or code with specific function signatures. You can generate targeted examples for any of these at scale, filling the gaps that natural data cannot cover at any realistic crawl budget.

A third reason is label quality. In natural datasets, labels are often noisy or require expensive human annotation. A capable teacher model can generate both the input and the ideal output, effectively providing its own supervision signal. This is particularly powerful for instruction-following datasets, where the task is to train a model to respond helpfully to user instructions. Collecting 50,000 human-annotated instruction-response pairs requires hundreds of annotator hours. Generating 50,000 synthetic instruction-response pairs from a capable teacher requires a few API calls and a modest budget.

A fourth reason, sometimes overlooked, is controllability. Natural data reflects what happened to be written and published; synthetic data reflects what you deliberately construct. If a model is weak at a particular reasoning pattern, you can build a seed set that generates thousands of examples targeting that exact weakness. If a model overuses hedging language, you can generate training examples that demonstrate confident, direct answers. This degree of intentional shaping is impossible with natural data.

Building on the data quality pipeline assembled across Part XX, synthetic data is the final tool in the data curation toolkit: after deduplication (MinHash), quality filtering, toxicity removal, PII scrubbing, and careful data mixing, synthetic data fills the remaining gaps with purpose-built examples.