Knowledge Distillation › Summary

Knowledge distillation transfers a large teacher model's learned knowledge to a smaller student model through soft label supervision. The core mechanism is simple: train the student to match the teacher's output probability distribution rather than just the correct class label. This soft target carries structural information about which outputs are similar and which are different, giving a richer training signal than hard labels alone.

Temperature scaling is the key technique for amplifying this information. By dividing the teacher's logits by a temperature $\tau > 1$ before applying softmax, the near-zero probabilities for non-predicted outputs become large enough to compare, revealing the teacher's implicit knowledge about relationships between outputs. The same temperature is applied to the student during training and removed at inference. Higher temperatures expose more dark knowledge but can introduce noise if taken too far; the range $[2, 8]$ works well for most tasks.

The combined distillation objective mixes the distillation loss (cross-entropy against the teacher's soft labels) with the standard supervised loss. The mixing coefficient $\alpha$ and the temperature $\tau$ are the two primary hyperparameters. The $\tau^2$ scaling factor compensates for the gradient magnitude reduction that temperature scaling introduces, keeping the distillation gradient stable as temperature varies.

Teacher selection matters significantly. The teacher should substantially outperform the student-capacity baseline, be well-calibrated, and operate on the same data distribution. Very large capacity gaps benefit from progressive distillation through intermediate models. Ensemble teachers produce smoother soft labels by averaging the outputs of multiple models, reducing the idiosyncratic biases of any single training run.

For large language models, distillation operates in two regimes. Token-level distillation matches the teacher's per-token probability distributions, as in DistilBERT and TinyBERT, giving the richest signal but requiring access to the teacher's logits. Sequence-level distillation trains on teacher-generated text, as in Alpaca and Vicuna, and works even with closed-source teachers but loses the detailed probability information. On-policy distillation addresses the distribution shift problem by computing teacher guidance on the student's own generated sequences, at higher cost but with better alignment for generation tasks.

The next chapter on distillation variants explores feature distillation, attention transfer, and other techniques that extend beyond output-level supervision to transfer richer internal structure from teacher to student.