Loss Functions › Loss Function Selection for NLP Tasks

Choosing the right loss function is about matching the loss to the structure of the task. Here is a practical guide for common NLP scenarios.

Text classification (spam detection, sentiment, topic): Use categorical cross-entropy for single-label, binary cross-entropy per class for multi-label. Label smoothing (0.1) is a low-cost improvement for calibration.

Named entity recognition (NER): Token-level classification. Use categorical cross-entropy with class weights or focal loss if entity classes are rare compared to non-entity tokens. Most production NER systems deal with severe label imbalance because most tokens are not entities.

Language modeling (next-token prediction): Categorical cross-entropy on the vocabulary distribution. The large vocabulary ($K = 50{,}000+$) requires efficient computation. Label smoothing is sometimes applied at the vocabulary level.

Sequence-to-sequence tasks (translation, summarization): Cross-entropy on each output token, summed or averaged across the sequence. Label smoothing is standard.

Sentence similarity and semantic search: Triplet loss or InfoNCE. SimCSE and similar approaches use a form of noise-contrastive estimation that generalizes contrastive loss to multiple in-batch negatives.

Knowledge distillation: KL divergence between student and teacher distributions, often with temperature scaling to soften teacher outputs.

Regression tasks (predicting scores, ranking): MSE for clean data, Huber loss when outliers are present.