Loss Functions › Temperature Scaling

Temperature scaling modifies a model's logits before computing probabilities, controlling the sharpness of the predicted distribution. A temperature $T > 0$ is applied as:

where:

• $z_k$: the raw logit for class $k$
• $T$: the temperature parameter
• $p_k$: the temperature-scaled probability for class $k$

When $T = 1$, the standard softmax is recovered. When $T < 1$ (low temperature), the distribution becomes sharper: the class with the highest logit gets even more probability mass. When $T > 1$ (high temperature), the distribution becomes softer, approaching a uniform distribution as $T \to \infty$.

Temperature scaling is used in several contexts:

• Knowledge distillation: A high temperature (e.g., $T = 5$ or $T = 10$) is applied to the teacher's logits during training. This produces soft targets that reveal the relative similarity between classes. This provides richer supervision than hard one-hot labels.
• Calibration: After training, temperature can be tuned on a validation set to make the model better calibrated, without changing its predictions (only confidence scores change).
• Sampling from language models: During text generation, temperature controls the randomness of token sampling. Low temperature makes the model more deterministic, high temperature makes it more creative.

Post-training temperature scaling is notable for being a uniquely simple and effective calibration technique. Unlike platt scaling or isotonic regression, temperature scaling has only one parameter to tune, avoids overfitting on the calibration set, and does not change the model's accuracy on any task (only its confidence scores). The procedure is: freeze the model, search over values of $T$ to minimize the NLL on a held-out validation set, and apply the optimal $T$ at inference time.