Loss Functions › Summary

Loss functions are the optimization targets that shape everything about how a neural network trains. The key takeaways are:

• MSE penalizes large errors quadratically and works well for regression with clean data. It arises as maximum likelihood under a Gaussian noise assumption. Huber loss is MSE's more robust variant that transitions to linear behavior for large errors, preventing outliers from dominating the gradient.
• Cross-entropy is derived from maximum likelihood and is the standard loss for classification. It strongly penalizes confident wrong predictions via the logarithm. The gradient is simply the prediction error: $p_k - y_k$ for each class. Binary cross-entropy is the two-class special case, and extends naturally to multi-label classification.
• Weighted cross-entropy addresses class imbalance by multiplying each example's loss by the inverse frequency of its class. This ensures rare classes receive proportionally more training signal.
• Label smoothing prevents overconfidence by replacing one-hot labels with soft targets. A small epsilon (0.1) is a reliable improvement for most classification tasks, reducing the logit gap and improving calibration.
• Focal loss addresses class imbalance by dynamically down-weighting easy examples through a modulating factor $(1 - p)^\gamma$. The focusing parameter $\gamma = 2$ is a common default. Unlike class weighting, focal loss adapts to the model's current state rather than applying a fixed multiplier.
• KL divergence measures how far one distribution is from another and is the theoretical foundation linking cross-entropy to maximum likelihood estimation. It appears explicitly in distillation and variational methods.
• Temperature scaling controls the sharpness of a softmax distribution, used in distillation (high temperature), sampling (variable temperature), and calibration (post-hoc temperature search).
• Language modeling loss is cross-entropy applied at each token position over a large vocabulary, and perplexity is its exponential. This provides an interpretable measure of prediction quality.
• Contrastive and triplet losses enable representation learning without predefined class labels, forming the basis for modern sentence embedding systems. InfoNCE generalizes these to use the full batch as negatives, and is the foundation for models like SimCSE and DPR.

The next chapter covers backpropagation, which explains how the gradient of any of these loss functions flows backward through a neural network to update every parameter simultaneously. Understanding the loss is only half the picture. Backpropagation closes the loop between loss values and parameter updates.