Loss Functions › Limitations and Practical Considerations

Every loss function has failure modes, implementation pitfalls, and scenarios where it is the wrong choice. Understanding these is as important as understanding the formulas. This section covers the most common issues practitioners encounter when working with neural network losses in NLP.

Cross-entropy is a strong default, but it has failure modes. In imbalanced classification settings (which are more common in NLP than often appreciated: most sentences are not toxic, most tokens are not named entities, most documents are not spam), the model learns to predict the majority class. The loss gets dominated by the many easy negatives. Focal loss and class weighting both address this, but they add hyperparameters that need tuning.

Label smoothing helps with calibration but does not help with imbalance. These are two separate problems with separate solutions. Do not conflate them. A model can be well-calibrated but still fail on rare classes, and a model can handle rare classes well but still be overconfident in its probability estimates.

When dealing with severe imbalance, it is worth combining multiple approaches: class-weighted loss handles the frequency imbalance statically, while focal loss handles it dynamically based on current model confidence. Using both together can be more effective than either alone.

A common error is using MSE as a loss for classification by treating class probabilities as continuous targets. MSE does not respect the constraint that probabilities must sum to 1, and it does not penalize confident wrong predictions as sharply as cross-entropy. Cross-entropy is the principled choice for classification because it is derived from maximum likelihood under the categorical distribution. Using MSE for classification is like assuming the targets follow a Gaussian distribution when they are categorical, which produces suboptimal gradients and slow convergence.

There are edge cases where regression-style losses are used on probability-like outputs. Semantic similarity tasks sometimes predict a score between 0 and 1 representing similarity, where MSE or Huber loss is reasonable. But this is because the underlying task is a regression task (predicting a real-valued similarity score), not a classification task (predicting a discrete label).

Binary cross-entropy applied directly to sigmoid outputs can cause gradient vanishing. When the sigmoid is nearly saturated (sigmoid output close to 0 or 1), its derivative is nearly zero, and the chain rule multiplies this small derivative into the gradient. The gradient can become so small that the optimizer makes essentially no progress.

The solution is to use F.binary_cross_entropy_with_logits() instead of computing the sigmoid first and then applying BCE. The combined function uses the numerically stable formulation:

$$\mathcal{L}_{\text{BCE}} = \max(z, 0) - z \cdot y + \log(1 + e^{-|z|})$$

where $z$ is the logit before sigmoid. This avoids computing the exponential of large positive numbers and maintains numerical precision across the full range of logit values. Always pass raw logits to combined loss functions rather than pre-activating them.

Triplet loss fails silently in practice when using random triplets: most randomly sampled triplets are "easy" (already satisfying the margin constraint), so the loss is zero and no gradient flows. The model does not improve. Hard negative mining selects the hardest negative for each anchor (the negative that is currently closest to the anchor in embedding space), and semi-hard mining uses negatives that are close but still beyond the margin. This mining step is critical to making triplet loss work.

The computational cost of hard mining scales as $O(N^2)$ in the number of examples: for each anchor, you need to find the hardest negative among all other examples. In practice, online hard mining within a batch (treating all non-positive examples in the batch as potential negatives and selecting the hardest) is the standard approach.

The loss you train on should be related to the metric you evaluate on. Training with cross-entropy but evaluating with F1 score is common and often works well in practice. But when there is a significant mismatch, careful thought is needed. If you care about ranking (e.g., information retrieval), a pairwise or listwise ranking loss is more directly aligned with the evaluation metric than cross-entropy. If you care about precision at a specific recall threshold, focal loss with tuned $\alpha$ and $\gamma$ may align better with your deployment requirements.

The deeper principle here is that the loss function defines what the model considers an "error" and how much that error matters. If your deployment scenario weights certain errors (e.g., false negatives in medical diagnosis) much more heavily than others, your training loss should reflect this asymmetry. Standard symmetric losses will produce models optimized for average performance, which may not match the risk profile of your application.

Loss functions that involve logarithms or exponentials are vulnerable to numerical overflow and underflow. Key rules of thumb:

• Always use F.cross_entropy() with raw logits, not F.nll_loss() after manually applying F.softmax().
• Always use F.binary_cross_entropy_with_logits(), not F.binary_cross_entropy() after applying sigmoid.
• When implementing custom losses, use torch.clamp() to keep log arguments away from zero, or use PyTorch's numerically stable built-in functions where possible.
• Watch for NaN gradients: they often signal a loss computation that hits a numerical singularity. Log(0) and 0/0 are the most common causes.

Numerical stability is not an academic concern. Production models trained on large corpora with billions of parameter updates can fail silently if any loss computation produces NaN, as NaN propagates through the network and corrupts all weights. Defensive programming with stable loss implementations is essential.