Loss Functions › Label Smoothing

Standard cross-entropy trains the model to push probabilities toward 0 or 1 for the correct and incorrect classes. This works well, but it can cause overconfidence: the model may assign extremely high probability to one class even for ambiguous examples, becoming poorly calibrated.

Label smoothing addresses this by replacing the hard one-hot labels with soft labels that reserve a small probability mass $\epsilon$ for all classes:

$$y_k^{\text{smooth}} = \begin{cases} 1 - \epsilon + \frac{\epsilon}{K} & \text{if } k = y^* \\ \frac{\epsilon}{K} & \text{if } k \neq y^* \end{cases}$$

where:

• $\epsilon$: the smoothing parameter, typically 0.1
• $K$: the total number of classes
• $y^*$: the index of the true class
• $1 - \epsilon + \frac{\epsilon}{K}$: the smoothed probability for the correct class (slightly less than 1)
• $\frac{\epsilon}{K}$: the smoothed probability for each incorrect class (slightly above 0)

The cross-entropy loss with smoothed labels becomes:

$$\mathcal{L}_{\text{LS}} = -(1 - \epsilon) \log p_{y^*} - \frac{\epsilon}{K} \sum_{k=1}^{K} \log p_k$$

where:

• $p_{y^*}$: the predicted probability for the true class
• $\sum_{k=1}^{K} \log p_k$: the sum of log probabilities across all classes

Label smoothing has two effects. It discourages the model from assigning zero probability to any class, which improves calibration. It also reduces the gap in logits between the correct class and incorrect classes, making the model's representations more spread out. The original Transformer paper used $\epsilon = 0.1$, and label smoothing has since become standard in many NLP models.

To see why label smoothing improves calibration, consider what happens during training without it. The optimal model under standard cross-entropy would assign probability 1 to the true class and probability 0 to all other classes, for every training example. This means the logits for the true class would need to go to infinity, which is impossible in finite training but the model tries to approach. This drives the logit gap (difference between the top logit and the second-highest) to be very large, leading to overconfident predictions. Label smoothing provides a finite target: the optimal model assigns $1 - \epsilon + \epsilon/K$ to the true class, which requires a finite logit gap. Training naturally converges to this finite optimal, producing better-calibrated confidence scores.