Synthetic Data › Knowledge Distillation

Knowledge distillation is the process of transferring the capability of a large, expensive teacher model into a smaller, efficient student model. The fundamental motivation is economic: large models are expensive to run at inference time. If a 7B student model can achieve 90% of a 70B teacher's performance on your tasks, deploying the student saves roughly 10x in inference compute. Synthetic data is the primary mechanism for distillation in modern LLM training: the teacher generates training examples, and the student trains on those examples.

The connection between synthetic data generation and distillation is direct. When you call GPT-4 to generate instruction-following examples and train a smaller model on them, you are doing response distillation. The student is learning to produce responses in the style and at the quality level of the teacher, compressing the teacher's learned behavior into a smaller set of parameters.

The simplest form of distillation is response distillation: the teacher generates complete responses to input prompts, and the student learns to mimic those responses via standard supervised fine-tuning.

Given a dataset of prompts $\mathcal{D} = \{x_1, x_2, \ldots, x_N\}$, the teacher model $T$ generates a response $y_i = T(x_i)$ for each prompt. The student model $S$ is then trained to minimize the negative log-likelihood of the teacher's response tokens, given the prompt and all preceding tokens. This is the standard next-token prediction loss, applied to teacher-generated text:

$$\mathcal{L}_{\text{response}} = -\sum_{i=1}^{N} \sum_{t=1}^{|y_i|} \log P_S(y_i^t \mid x_i, y_i^{<t})$$

where:

• $N$ is the number of training examples in $\mathcal{D}$
• $x_i$ is the input prompt for example $i$
• $y_i$ is the teacher-generated response for prompt $i$, consisting of $|y_i|$ tokens
• $y_i^t$ is the $t$-th token of response $y_i$, serving as the prediction target at position $t$
• $y_i^{<t}$ is the sequence of response tokens at positions before $t$ (the left context)
• $P_S(y_i^t \mid x_i, y_i^{<t})$ is the student's predicted probability for the correct token at position $t$

The outer sum over $i$ accumulates the loss across all examples. The inner sum over $t$ accumulates the token-level loss within a single response. Minimizing this objective encourages the student to assign high probability to exactly the sequence of tokens the teacher produced.

This is identical to standard language model fine-tuning, but the labels come from the teacher rather than human annotators. Response distillation is what most instruction-tuning pipelines do when they call GPT-4 to generate training data. The simplicity is its virtue: it requires no special training infrastructure, no access to the teacher's internal activations, and no modification to the standard fine-tuning pipeline.

A more powerful form of distillation transfers the reasoning process along with the final answer. Chain-of-thought (CoT) distillation generates step-by-step reasoning traces from the teacher and trains the student to reproduce those traces.

The Orca paper demonstrated that training a 13B parameter model on detailed reasoning traces from GPT-4 produced a student that substantially outperformed models trained on final answers alone. The reasoning traces provide explicit supervision for the intermediate steps that the model needs to solve complex problems, rather than supervising only the endpoint. The student learns the answer and the path for arriving at it.

Generating reasoning chains requires prompting the teacher carefully. System messages like "Think step by step and explain your reasoning in detail before giving the final answer" elicit chain-of-thought outputs. The generated chains then serve as training targets. The key insight is that chain-of-thought prompting was already known to improve reasoning at inference time; distillation makes that reasoning capability a trained property of the student rather than something that requires special prompting at inference time.

The quality of distilled reasoning chains depends on the quality of the teacher's chains. A teacher that produces crisp, well-structured reasoning traces produces better training data than a teacher whose reasoning is scattered or contains errors. For this reason, reasoning chain distillation is most effective with the strongest available teacher models, and the quality investment in the teacher pays off disproportionately in the student's downstream performance.

One important practical decision is whether to train the student to produce reasoning traces at inference time, or to use the reasoning traces only as intermediate supervision during training. The first approach (having the student reason aloud at inference time) produces more interpretable models and generally achieves better performance on complex tasks. The second approach produces a more compact model that does not need to generate reasoning tokens, which reduces inference latency. The choice depends on whether the deployment context values interpretability or speed more.

Classical knowledge distillation (from Hinton et al., 2015) transfers the teacher's answers and the teacher's soft probability distribution over the next token. Instead of training the student on the argmax output of the teacher, the student is trained to match the teacher's full output distribution.

The key idea is that a teacher model's probability distribution over tokens carries much more information than its single best prediction. When a teacher assigns 60% probability to token A, 30% to token B, and 10% to token C, it is communicating that A and B are both plausible, and that B is more plausible than C. Training on this soft distribution teaches the student the full structure of the teacher's knowledge about token plausibility, not just which token happened to be the argmax.

The distillation loss combines the student's cross-entropy loss on the correct labels with a divergence loss against the teacher's distribution:

$$\mathcal{L}_{\text{distill}} = (1 - \alpha) \mathcal{L}_{\text{CE}}(y, P_S) + \alpha \cdot T^2 \cdot \text{KL}(P_T^{(T)} \parallel P_S^{(T)})$$

where:

• $\mathcal{L}_{\text{CE}}$ is the standard cross-entropy loss against the ground truth label $y$
• $P_T^{(T)}$ is the teacher's softmax distribution at temperature $T$
• $P_S^{(T)}$ is the student's softmax distribution at temperature $T$
• $\alpha$ is a mixing coefficient controlling the balance between hard labels and soft targets
• $T$ is the temperature parameter that softens both distributions

The temperature $T > 1$ softens the distributions, spreading probability mass from the argmax token to non-argmax tokens. This reveals the teacher's "dark knowledge": its beliefs about which wrong answers are more plausible than others. A teacher that assigns 40% probability to token A and 35% to token B is conveying that A and B are nearly interchangeable in this context, which is much more information than a teacher that assigns 100% to A and nothing to B.

The $T^2$ scaling in the loss formula compensates for a mathematical artifact of temperature scaling. When you divide logits by $T$ before softmax, the gradients of the resulting KL divergence are also scaled by $1/T^2$. The $T^2$ multiplier in the loss cancels this scaling, making sure that the soft loss and the hard loss contribute comparable gradient magnitudes regardless of the temperature choice.

Logit distillation requires access to the teacher model's logit outputs, which is not available when using API-only models like GPT-4. It is most applicable when you own or have white-box access to the teacher model. For open-source teacher models (Llama, Mistral, and so on), logit distillation is straightforward to implement. For proprietary API teachers, response distillation is the only available option.

Speculative decoding is an inference efficiency technique that has a natural connection to distillation. A small draft model proposes token sequences, and the large verifier model either accepts or rejects them in parallel. Tokens accepted by the verifier match the verifier's distribution; tokens rejected are replaced.

The data generated during speculative decoding contains rich alignment signals: each accepted draft token is one where the small model's prediction aligned with the large model's. Collecting these accepted tokens across many inference runs produces a training dataset for improving the draft model, making it gradually more aligned with the verifier. This is an organic form of online distillation, where the distillation signal comes from the natural operation of the inference system rather than from a separate training data generation stage.

The appeal of this approach is that it turns inference compute into training data without requiring any additional API calls or prompt construction. Every deployment of a speculative decoding system automatically generates a stream of alignment data that can be used to improve the draft model, creating a flywheel where deployment improves the system over time.

Distillation works best when the teacher-student capacity gap is moderate. If the teacher is GPT-4 and the student is a 70B model, distillation can transfer substantial capability because the student has enough representational capacity to capture most of the teacher's behavior. If the student is a 1B model trying to learn from GPT-4's outputs, the gap may be too large: the student's representational capacity cannot fit the full range of teacher behaviors.

The mathematical intuition is that a neural network can only represent functions within its function class, and smaller networks have smaller function classes. No matter how many examples you provide from a teacher whose behavior lies outside that function class, the student cannot represent the teacher's full behavior. The student will find the best approximation it can, but there is a hard upper bound on how closely it can approximate the teacher.

In practice, this manifests as the student producing grammatically correct but factually shallow responses. The student has learned the style of the teacher's outputs (confident, well-structured, appropriately hedged) without acquiring the underlying knowledge. This is sometimes called style distillation as opposed to capability distillation. A student that has absorbed the teacher's stylistic patterns but not its knowledge will sound authoritative while being factually unreliable, which can be worse than a student that sounds uncertain, because users may trust the confident-sounding incorrect answer.

Mitigation strategies include:

• Using smaller teacher models that are closer in capability to the student, so the capacity gap is more bridgeable
• Curriculum learning: start with simpler examples from the teacher and gradually increase complexity as the student improves
• Filtering the teacher's outputs to keep only examples within the student's current competence range, avoiding examples so far above the student's level that they produce only noise in the gradient signal
• Combining distillation with pretraining on natural data, which provides the foundational knowledge the student needs before it can make sense of the teacher's outputs