Knowledge Distillation › Limitations and Practical Considerations

Distillation is powerful, but it is not a universal solution to the model compression problem. Understanding its limitations is essential for applying it correctly and for knowing when to reach for a different technique.

The capacity gap problem is the most fundamental limitation. When the teacher is much larger than the student, the student cannot represent the teacher's distribution within its parameter budget. In these regimes, distillation can hurt: the student wastes capacity trying to approximate distributions it cannot represent, rather than learning the best possible solution within its own capacity. Empirical evidence suggests that the distillation benefit diminishes sharply when the parameter ratio exceeds roughly 10:1. This does not mean distillation fails entirely at large ratios, but it does mean the gains are smaller and less predictable, and progressive distillation is often necessary to realize the full potential.

Distillation also assumes the teacher is reliably better than the student. If the teacher makes confident errors, the student learns those errors as well, and the soft labels propagate misinformation. This is particularly problematic in domains where the teacher has limited training data, where it extrapolates far beyond its training distribution, or where it has developed systematic biases. In these cases, the distillation signal is noisier than the ground-truth labels, and a lower value of $\alpha$ is appropriate. There is also no mechanism in standard distillation for the student to recognize when the teacher is wrong: the student simply follows the teacher's distribution, errors and all.

The computational cost of distillation is often underestimated. Training the student requires the teacher's logits or soft labels at every training step, which means either keeping the teacher loaded in memory alongside the student (doubling the memory requirement), pre-computing and storing the teacher's distributions over the entire training corpus (requiring enormous storage), or making online API calls to a hosted teacher (adding latency and cost). For very large training datasets, none of these approaches is trivially cheap. The decision to use distillation should include an honest accounting of these costs.

The data distribution assumption is another limitation that practitioners frequently overlook. Distillation works best when teacher and student operate on the same data distribution. If the deployment distribution differs significantly from the training distribution, both teacher and student may fail, and the student inherits the teacher's failure modes rather than learning to handle distribution shift. Out-of-distribution generalization is not automatically transferred through distillation. It must be built into the training data or addressed through other techniques like data augmentation or distribution-aware training objectives.

Despite these limitations, distillation remains one of the most effective techniques for creating compact, deployable models. DistilBERT retains 97% of BERT's accuracy at 40% fewer parameters and 60% faster inference. TinyBERT achieves competitive performance with roughly 7x fewer parameters. Smaller instruction-following models like Mistral-7B and Phi-2 demonstrate that thoughtful distillation and data curation can produce models that punch far above their weight class. The Phi-2 model from Microsoft, with only 2.7B parameters, achieved performance on reasoning benchmarks comparable to models with five to ten times as many parameters, largely because of careful distillation-informed training data curation.

The practical takeaway is this: distillation works best when the capacity gap is moderate (2-10x), the teacher is well-calibrated, the data distribution is consistent, and the training pipeline can accommodate the teacher's memory or compute requirements. When these conditions are met, distillation reliably produces small models that outperform any alternative training strategy at the same parameter count.