Loss Functions › Focal Loss for Class Imbalance

Many real-world classification problems have severe class imbalance: fraud detection, rare disease diagnosis, named entity recognition (where most tokens are not entities). Standard cross-entropy performs poorly in these settings because easy negative examples (confidently classified background tokens) dominate the loss, overwhelming the contribution of the rare positive class.

The problem is more subtle than just having fewer positive examples. Even after you account for class frequency, the model tends to become good at classifying the easy majority class early in training. Once the model can classify the majority class with high confidence, those examples contribute almost no loss (because $-\log(0.99) \approx 0.01$), but there are many of them. The rare, hard-to-classify minority examples contribute more loss per example, but there are so few of them that they still get drowned out.

Focal loss (introduced by Lin et al., 2017 in the context of object detection) down-weights easy examples dynamically, focusing training on hard examples:

$$\mathcal{L}_{\text{focal}} = -(1 - p_{y^*})^\gamma \log p_{y^*}$$

where:

• $p_{y^*}$: the model's predicted probability for the true class
• $\gamma \geq 0$: the focusing parameter (typically 2.0)
• $(1 - p_{y^*})^\gamma$: the modulating factor, close to 1 for low confidence and close to 0 for high confidence

When the model is already confident about an example ($p_{y^*} \approx 0.9$), the modulating factor $(1 - 0.9)^2 = 0.01$ reduces the loss contribution by 100x. When the model is uncertain ($p_{y^*} \approx 0.5$), the factor $(1 - 0.5)^2 = 0.25$ reduces the loss by only 4x. Hard examples (where the model is wrong or uncertain) contribute more to the loss than easy examples.

Focal loss is often combined with a class-weighting factor $\alpha$:

$$\mathcal{L}_{\text{focal}} = -\alpha_{y^*} (1 - p_{y^*})^\gamma \log p_{y^*}$$

where:

• $\alpha_{y^*}$: the per-class weight, which can be set to the inverse frequency of each class to further up-weight rare classes

Setting $\gamma = 0$ recovers standard cross-entropy. A common default is $\gamma = 2$ and $\alpha = 0.25$ for the positive class.

Focal loss adapts dynamically to each example based on the model's current confidence, rather than applying a fixed multiplier like class weighting. This is a powerful property: early in training when the model is uncertain about everything, focal loss behaves similarly to standard cross-entropy. As training progresses and the model masters easy examples, focal loss automatically down-weights those contributions and concentrates the learning signal on the difficult cases that remain.