Loss Functions › Worked Example: Loss Surface for a 3-Class Problem

To build intuition for how different losses behave, consider a concrete scenario. A model is classifying text into three sentiment categories: negative (class 0), neutral (class 1), and positive (class 2). The model produces raw logits for each class, and we want to see how the loss changes as the model's confidence changes.

Consider a single example with true label "positive" (class 2). The model outputs logit vectors of different shapes:

• Correct and confident: logits $[0.5, 0.5, 3.0]$. After softmax: approximately $[0.05, 0.05, 0.90]$. Cross-entropy loss: $-\log(0.90) \approx 0.10$.
• Uncertain: logits $[1.0, 1.0, 1.0]$. After softmax: $[0.33, 0.33, 0.33]$. Cross-entropy loss: $-\log(0.33) \approx 1.10$.
• Wrong and confident: logits $[3.0, 0.5, 0.5]$. After softmax: approximately $[0.90, 0.05, 0.05]$. Cross-entropy loss: $-\log(0.05) \approx 3.0$.

The loss spans from 0.10 to 3.0 across these three scenarios. This provides a strong gradient signal that distinguishes confident-correct from uncertain from confident-wrong. This is the key property of cross-entropy: it does not just measure whether the prediction is right or wrong, it measures the quality of the confidence estimate.

Now compare focal loss with $\gamma = 2$:

• Correct and confident: $-\log(0.90) \times (1 - 0.90)^2 = 0.10 \times 0.01 = 0.001$.
• Uncertain: $-\log(0.33) \times (1 - 0.33)^2 = 1.10 \times 0.45 = 0.50$.
• Wrong and confident: $-\log(0.05) \times (1 - 0.05)^2 = 3.0 \times 0.90 = 2.70$.

The confident-correct loss drops from 0.10 to 0.001 (100x reduction), while the confident-wrong loss barely changes from 3.0 to 2.70. Focal loss dynamically concentrates the training signal on the hard cases while suppressing easy ones.