Knowledge Distillation › Teacher Selection

The success of distillation depends critically on choosing the right teacher model. A bad teacher teaches bad habits. A teacher that is too different from what the student can learn from provides targets the student cannot approximate. Getting teacher selection right is often more important than fine-tuning the temperature or the mixing coefficient.

Several properties characterize a good distillation teacher:

• High accuracy on the target task: The teacher should be substantially better than a student trained without distillation. If the teacher and student have similar accuracy, distillation provides little benefit because the soft labels are not much more informative than what the student could generate on its own.
• Well-calibrated probabilities: The teacher's soft labels should encode meaningful uncertainty. An overconfident teacher that assigns 99.99% probability to all its predictions produces soft labels that are barely softer than hard labels, even with temperature scaling. Calibration can be measured by comparing the teacher's confidence to its actual accuracy on a held-out set.
• Same task and domain: The teacher and student should operate on the same input distribution. A teacher trained on formal news text distills poorly to a student meant for conversational text because the probability distributions over tokens reflect very different stylistic patterns.
• Compatible output space: Teacher and student must share the same vocabulary and output format for token-level distillation to work directly. Distilling across vocabularies requires additional alignment steps.

Beyond these requirements, a good teacher has also "learned the right things" in the sense that its internal representations are organized in a way that reflects linguistic or semantic structure, not artifacts of training noise. This is hard to measure directly, but models trained on more data and with more compute tend to have more structured internal representations.

A single model's soft labels reflect its particular training trajectory and may encode idiosyncratic biases. One teacher might assign systematically high probability to certain tokens due to artifacts in its pretraining data. Another might have learned slightly different probability assignments for ambiguous cases based on the random initialization it started from. These idiosyncrasies add noise to the distillation signal.

Using an ensemble of models as the teacher, averaging their output distributions, produces smoother and more reliable soft labels:

$$p^T_i = \frac{1}{M} \sum_{m=1}^{M} p^{T_m}_i$$

where:

• $M$: the number of teacher models in the ensemble
• $p^{T_m}_i$: the probability assigned to token $i$ by the $m$-th teacher

Ensemble distillation is especially effective when the individual teachers make different types of errors. Their averaged distribution is more informative because regions of uncertainty in one model are compensated by confidence in another. The student learns from a "collective wisdom" that no single model could provide. This is one of the original motivations in Hinton et al.'s paper: they noted that the ensemble provides a more stable and less biased training target than any individual model.

In practice, maintaining and querying an ensemble of large models is expensive, but even a small ensemble of two or three models substantially improves the quality of the soft labels. When the teachers are smaller models that fit easily in GPU memory, ensemble distillation is worth the additional compute.

A subtle but important issue is the capacity gap: when the student is much smaller than the teacher, it may not be able to approximate the teacher's distribution even in principle. A student with 10 layers trying to match a 96-layer teacher faces a learning problem that may be fundamentally unsolvable within its parameter budget.

The student can only represent distributions that are consistent with its own architecture. A very small model has limited expressivity. Its softmax output is constrained to distributions that its particular arrangement of weights can produce. If the teacher's distribution requires a more complex functional form than the student can represent, the student cannot match it regardless of how long it trains.

Empirically, very large capacity gaps lead to poor distillation. The student can learn that certain tokens are more likely than others, but it cannot reproduce the fine-grained probability differences that make the teacher's soft labels informative. Worse, trying to match an unachievable target can hurt the student's training, leading to worse performance than training without distillation. The student wastes optimization effort pursuing targets it cannot reach, rather than learning the best possible solution within its own capacity.

The practical solution is to use an intermediate "assistant" or "mediator" model that bridges the capacity gap. The original large teacher first distills into a medium-sized model, which then distills into the target small model. This progressive distillation produces students that outperform those trained directly from the large teacher when the gap is extreme. The intermediate student learns to represent a simplified version of the teacher's distribution, and this simplified version is much easier for the final small student to approximate.