Loss Functions › What Makes a Good Loss Function

A loss function $\mathcal{L}$ takes the model's predicted output $\hat{y}$ and the true target $y$, and returns a non-negative scalar indicating how far apart they are. Before diving into specific losses, it is worth spending time on the properties that make one loss function better suited to a problem than another. Three properties matter most:

• Differentiability: The optimizer needs to compute gradients of the loss with respect to every model parameter. Non-differentiable losses can stall training. In practice, some losses (like MAE) are non-differentiable at exactly one point, and this is handled using subgradients, but losses with large non-differentiable regions are problematic.
• Smoothness: Flat regions (zero gradient) prevent learning. Sharp discontinuities cause training instability. A well-designed loss function provides informative gradient signal across the full range of predictions, not just near the decision boundary.
• Alignment with task: The loss should penalize the errors that matter most for the task. A loss that treats all errors equally may produce models that fail on rare but important cases.

There is a fourth property that is less often stated explicitly but just as important: computational tractability. For language models with vocabularies of 50,000 or more tokens, computing a full softmax distribution over the entire vocabulary at every position of a long sequence is expensive. This computational pressure has driven the development of approximations like sampled softmax, noise contrastive estimation, and hierarchical softmax, all of which modify the loss to make it computable within a reasonable budget.

There is no universal best loss function. Regression problems, binary classification, multiclass classification, and sequence modeling each have losses that are better matched to their structure. The key insight is that every loss encodes assumptions about the shape of the problem: what errors are costly, how uncertainty should be handled, and what the output distribution looks like. When those assumptions match reality, training proceeds efficiently. When they do not, you get models that optimize the wrong objective.

Most loss functions are not invented from scratch. They are derived from the same statistical principle: maximum likelihood estimation (MLE). The idea is simple. Given a model with parameters $\theta$ and a dataset of examples, we want to find the parameters that make the observed data as probable as possible under the model. Concretely, if the model defines a probability distribution $p_\theta(\mathbf{y} | \mathbf{x})$ over outputs $\mathbf{y}$ given inputs $\mathbf{x}$, MLE finds:

$$\hat{\theta} = \arg\max_{\theta} \prod_{i=1}^{N} p_\theta(y_i | x_i)$$

Taking the logarithm converts the product to a sum and does not change the optimal solution because logarithm is a monotone function:

$$\hat{\theta} = \arg\max_{\theta} \sum_{i=1}^{N} \log p_\theta(y_i | x_i)$$

Negating this gives a minimization objective, which is what gradient descent needs:

$$\hat{\theta} = \arg\min_{\theta} -\sum_{i=1}^{N} \log p_\theta(y_i | x_i)$$

This is negative log-likelihood (NLL). Cross-entropy for classification and mean squared error for Gaussian regression are both special cases of NLL under different distributional assumptions. Cross-entropy arises when you model the output as a categorical distribution. MSE arises when you model the output as a Gaussian distribution with fixed variance. Understanding this connection means you understand why these specific formulas are used, not just how to apply them.