Knowledge Distillation › Temperature in Distillation

The most practically important concept in distillation is temperature scaling, and it is worth understanding in detail because it fundamentally changes what information the student receives. Without temperature scaling, distillation often provides little improvement over hard-label training, because the soft labels are too similar to hard labels to provide additional signal.

Language model outputs tend to be sharply peaked. Given a sentence like "The sky is ___", the model assigns very high probability to "blue" and near-zero probability to essentially everything else. When these near-zero probabilities are used as soft labels, they provide almost no additional information compared to hard labels: the distribution is essentially one-hot.

The teacher's knowledge about relationships between less likely tokens is compressed into these tiny probability values. Words that are semantically related to "blue" (like "clear," "gray," or "overcast") receive probabilities that are only slightly higher than completely unrelated words. At floating-point precision, these differences exist, but they are numerically tiny. A student trying to match these near-zero values receives a training signal that is essentially the same as trying to match zeros. The relational structure is there in principle, but it is buried beneath the noise floor of floating-point arithmetic.

This is the fundamental problem that temperature scaling solves. The information is present in the teacher's logits, but it is compressed in a way that makes it invisible to the training objective. We need to uncompress it.

Temperature scaling addresses this by flattening the distribution before using it as a soft label. The teacher's logits $z^T_i$ are divided by a temperature parameter $\tau$ before the softmax is applied:

$$p^T_i(\tau) = \frac{\exp(z^T_i / \tau)}{\sum_{j=1}^{V} \exp(z^T_j / \tau)}$$

where:

• $z^T_i$: the teacher's logit (pre-softmax score) for token $i$
• $\tau$: the temperature parameter ($\tau > 0$)
• $p^T_i(\tau)$: the temperature-scaled probability for token $i$

When $\tau = 1$, this is the standard softmax. When $\tau > 1$, the distribution becomes flatter: smaller probabilities grow relative to the largest ones. When $\tau < 1$, the distribution becomes sharper, concentrating even more mass on the peak.

To see why temperature matters concretely, consider logits $[10, 2, 1, 0]$. At $\tau = 1$, the softmax gives approximately $[0.9997, 0.0003, 0.0001, 0.00004]$: essentially a one-hot distribution. At $\tau = 4$, the same logits give roughly $[0.76, 0.13, 0.08, 0.05]$: a much softer distribution where the relative magnitudes of all four logits are visible in the probability values. The smaller logits now carry meaningful probability mass and can guide the student's learning.

The mechanics of why this works come from the exponential function. When logits are divided by a large $\tau$, the differences between them shrink. The exponential function maps small differences to values close to 1, so the ratio between the largest and smallest probabilities decreases. What was a thousand-fold difference at $\tau = 1$ becomes a moderate difference at $\tau = 4$, moving probabilities that were numerically invisible into ranges where the gradient signal is strong.

During distillation, the same temperature $\tau$ is applied to both the teacher's distribution (to produce soft labels) and the student's distribution (to compute the loss). The student is trained to match the teacher's soft output. After training is complete, the temperature is reset to $\tau = 1$ for inference. The student's weights have been shaped by exposure to the temperature-scaled distributions, but it produces predictions at the standard temperature.

Hinton et al. introduced the term "dark knowledge" to describe the information encoded in the small probabilities that standard training ignores. A teacher that assigns probability 0.0001 to "feline" when predicting the word "cat" is revealing a structural relationship between these two words. This relationship is invisible under hard-label training but becomes visible when the distribution is temperature-scaled.

The key insight is that the ratio of probabilities at high temperature reveals structure that absolute probabilities hide. If the teacher assigns logit 10 to "cat," logit 2 to "feline," and logit $-5$ to "airplane," the ratio $p(\text{feline}) / p(\text{airplane})$ encodes the semantic proximity of cat and feline relative to airplane. At low temperature, both probabilities round toward zero and the ratio becomes numerically unreliable. At high temperature, both probabilities are substantial and the ratio is preserved, giving the student a meaningful gradient signal that "feline" and "cat" are semantically related while "airplane" is not.

This is the distinction between distillation and a minor regularization variant. The teacher's logits contain a rich relational structure built up over training. Temperature scaling is the tool that makes this structure accessible to the student.

The right temperature depends on how peaked the teacher's distribution is and how much dark knowledge you want to expose. For tasks with very confident outputs (like clearly labeled image classification), moderate temperatures of $\tau = 2$ to $\tau = 4$ are usually sufficient to make the secondary probabilities visible. For tasks with ambiguous targets (like next-token prediction in language models), higher temperatures up to $\tau = 10$ can reveal richer structure, but there is a point of diminishing returns.

At very high temperatures, the distribution approaches a uniform distribution, and the distinctions between tokens blur. The teacher's guidance becomes noise rather than signal. At very low temperatures, the distribution collapses toward a one-hot vector and the soft labels carry no more information than hard labels. The sweet spot is in between: high enough that secondary probabilities are substantial, low enough that the teacher's ranking of outputs is still preserved.

A practical approach is to treat the temperature as a hyperparameter and tune it on a held-out validation set. In practice, values in the range $[2, 8]$ work well for most classification and language modeling tasks. For well-trained, high-confidence teachers, lean toward lower temperatures; for less confident or more uncertain teachers, lean toward higher temperatures.

When using temperature scaling during distillation, the gradients of the distillation loss are scaled by $1/\tau^2$ compared to the gradients of the standard supervised loss. This happens because the softmax function with temperature $\tau$ produces logit gradients that are $1/\tau$ times smaller than those of the standard softmax, and this factor appears twice in the computation: once for the teacher's distribution and once for the student's. The net effect is that increasing $\tau$ reduces the magnitude of the distillation gradient by a factor of $\tau^2$.

To keep the distillation loss contribution stable relative to the supervised loss when varying $\tau$, Hinton et al. recommend multiplying the distillation loss term by $\tau^2$:

$$\mathcal{L} = \alpha \cdot \tau^2 \cdot \mathcal{L}_{\text{distill}}(\tau) + (1 - \alpha) \cdot \mathcal{L}_{\text{CE}}$$

where $\mathcal{L}_{\text{distill}}(\tau)$ denotes the distillation loss computed with temperature $\tau$. The $\tau^2$ factor compensates for the gradient shrinkage caused by the temperature scaling, making sure that the magnitude of the distillation gradient remains roughly constant as $\tau$ varies. This makes it easier to tune $\alpha$ independently of $\tau$: you can increase the temperature to reveal more dark knowledge without accidentally reducing the effective learning rate on the distillation objective.

This $\tau^2$ correction is often omitted in implementations without dramatic consequences, especially when $\alpha$ is tuned empirically, but it simplifies hyperparameter tuning and is good practice when sweeping over temperature values.