Distillation Variants › Summary

This chapter extended knowledge distillation from output-level soft labels into four richer transfer mechanisms:

• Feature distillation aligns intermediate hidden representations between teacher and student using a projection layer, giving a dense training signal that guides the student's internal geometry toward the teacher's representational structure.
• Attention transfer teaches the student to replicate the teacher's attention weight distributions, transferring structural knowledge about which tokens matter and how information flows through the network across all sequence positions.
• Progressive distillation chains multiple small compression steps to avoid the sharp quality degradation that occurs when teacher and student differ by a large capacity factor in a single step, motivated by the rate-distortion observation that smaller per-step capacity gaps compose more favorably than one large gap.
• On-policy distillation trains the student on sequences it generates itself, eliminating the exposure bias that causes standard distillation to fail at inference time when the student generates its own off-distribution text. The choice between forward KL (mode-covering, diverse) and reverse KL (mode-seeking, focused) shapes the generation characteristics of the trained student.

Each variant addresses a specific limitation of vanilla KD. Feature distillation and attention transfer enrich the training signal by exploiting internal structure. Progressive distillation makes compression less sensitive to large teacher-student gaps. On-policy distillation corrects distribution shift in generation tasks. In practice, these methods are often combined: production distillation pipelines for large language models typically use feature distillation together with on-policy training, and progressive staging is standard when the target model is more than 10 times smaller than the teacher.

The on-policy methods described here also connect directly to the broader alignment literature. RLHF (Reinforcement Learning from Human Feedback) can be understood as a generalization of on-policy distillation where the "teacher" signal comes from human preferences rather than a larger language model. The policy gradient machinery is the same; only the reward source changes. Understanding on-policy distillation therefore gives you a foundation for understanding a large class of post-training alignment techniques.

The next chapter examines pruning as a complementary compression technique. While distillation trains a new smaller model from scratch using teacher supervision, pruning removes unnecessary components from an existing model, giving a different route to the same goal of efficient inference.