Distillation Variants › Distillation Variants

Knowledge distillation, introduced in the previous chapter, compresses a large teacher model into a smaller student by matching softened probability outputs. That approach transfers what the teacher believes about each class. But the teacher knows far more than its final predictions. Inside its intermediate layers are rich geometric structures: feature maps that capture syntax, attention patterns that signal which words matter, and gradient dynamics that shape learning itself. Distillation variants extract these deeper signals, giving students access to knowledge that the output distribution alone cannot convey.

This chapter covers four extensions that push distillation beyond soft labels. Feature distillation transfers intermediate representations, aligning the hidden-layer geometry of student and teacher. Attention transfer teaches the student to replicate the teacher's attention patterns, guiding where it looks rather than just what it concludes. Progressive distillation takes a different angle, treating the teacher-student relationship as a staged refinement process that iteratively compresses models while preserving quality. On-policy distillation addresses a subtle but important failure mode of standard distillation: the mismatch between the distribution of inputs the student is trained on and the distribution it encounters at inference time.

Understanding these variants matters because each solves a different gap in vanilla output-level distillation. You will see that the choice of which variant to use depends on your task, your architecture, and how much computational budget you have for training. In practice, production pipelines rarely use a single method in isolation. The most capable compressed models combine feature distillation, attention transfer, and on-policy generation into unified training objectives, extracting every available signal from the teacher.