Loss Functions › Regression Losses

Regression losses measure the discrepancy between a continuous predicted value and a continuous target value. The two most common choices are mean squared error and mean absolute error, and their differences are significant.

Mean squared error (MSE) is the most widely used regression loss. Given a batch of $N$ training examples with predictions $\hat{y}_i$ and targets $y_i$, MSE computes the average squared difference:

$$\mathcal{L}_{\text{MSE}} = \frac{1}{N} \sum_{i=1}^{N} (\hat{y}_i - y_i)^2$$

where:

• $N$: the number of examples in the batch
• $\hat{y}_i$: the model's prediction for example $i$
• $y_i$: the true target value for example $i$
• $(\hat{y}_i - y_i)^2$: the squared error, always non-negative

The squaring serves two purposes. First, it makes every error positive, so over-predictions and under-predictions both contribute to the loss. Second, it penalizes large errors disproportionately. An error of 2 contributes 4 to the loss, while an error of 10 contributes 100. This makes MSE sensitive to outliers.

The gradient of MSE with respect to the prediction $\hat{y}_i$ is:

$$\frac{\partial \mathcal{L}_{\text{MSE}}}{\partial \hat{y}_i} = \frac{2}{N} (\hat{y}_i - y_i)$$

where:

• $\frac{\partial \mathcal{L}_{\text{MSE}}}{\partial \hat{y}_i}$: the gradient, pointing in the direction of steepest increase of the loss
• $(\hat{y}_i - y_i)$: the residual, positive when the prediction is too high, negative when too low

The gradient is proportional to the error magnitude. Large prediction errors produce large gradients, which means the optimizer takes bigger steps to correct big mistakes. This property is exactly what makes MSE both useful and risky: useful because big errors dominate training and get fixed quickly, risky because a single outlier can dominate the loss and pull the model away from fitting the majority of the data.

There is a probabilistic interpretation of MSE that ties it back to maximum likelihood. If you assume that each target value is sampled from a Gaussian distribution centered at the model's prediction, then:

$$p(y_i | \hat{y}_i) = \frac{1}{\sigma\sqrt{2\pi}} \exp\left(-\frac{(y_i - \hat{y}_i)^2}{2\sigma^2}\right)$$

The negative log-likelihood under this Gaussian assumption is proportional to $\sum_i (y_i - \hat{y}_i)^2$, which is exactly MSE. So MSE is the correct loss when you believe the noise in your targets is Gaussian, with equal variance everywhere. When that assumption is violated (heavy-tailed noise, heteroskedastic variance, or outliers), MSE is the wrong loss.

Mean absolute error (MAE) uses the absolute value instead of the square:

$$\mathcal{L}_{\text{MAE}} = \frac{1}{N} \sum_{i=1}^{N} |\hat{y}_i - y_i|$$

where:

• $|\hat{y}_i - y_i|$: the absolute difference between prediction and target, always non-negative

MAE treats all errors with equal weight per unit of magnitude. A prediction off by 2 contributes exactly twice as much as a prediction off by 1, regardless of scale. This makes MAE much more robust to outliers than MSE: an outlier with error 10 contributes 10 to the loss instead of 100.

The gradient of MAE is:

$$\frac{\partial \mathcal{L}_{\text{MAE}}}{\partial \hat{y}_i} = \frac{1}{N} \cdot \text{sign}(\hat{y}_i - y_i)$$

where:

• $\text{sign}(\hat{y}_i - y_i)$: returns $+1$ if the prediction is too high, $-1$ if too low, and $0$ if exact

The gradient is constant in magnitude (either $+1/N$ or $-1/N$ per example), which means the optimizer always takes the same-sized step regardless of error size. This is the key tradeoff: MAE is more less brittle to outliers, but converges more slowly because it does not amplify gradients for large errors. It is also technically non-differentiable at zero, though this is handled in practice by taking the subgradient (which returns 0 at exactly zero error).

The probabilistic counterpart to MAE is a Laplace distribution for the noise. If the noise follows a Laplace distribution rather than a Gaussian, negative log-likelihood gives MAE. The Laplace distribution has heavier tails than the Gaussian, meaning it assigns more probability to large errors. Choosing MAE over MSE is equivalent to assuming the data has Laplace noise rather than Gaussian noise: a decision that matters when your dataset contains outliers.

Huber loss (also called smooth L1 loss) is a hybrid that behaves like MSE for small errors and MAE for large errors:

$$\mathcal{L}_{\delta}(\hat{y}_i, y_i) = \begin{cases} \frac{1}{2}(\hat{y}_i - y_i)^2 & \text{if } |\hat{y}_i - y_i| \leq \delta \\ \delta \cdot |\hat{y}_i - y_i| - \frac{1}{2}\delta^2 & \text{if } |\hat{y}_i - y_i| > \delta \end{cases}$$

where:

• $\delta$: a threshold parameter that controls the transition point between quadratic and linear behavior
• $\frac{1}{2}(\hat{y}_i - y_i)^2$: the quadratic (MSE-like) component for small errors
• $\delta \cdot |\hat{y}_i - y_i| - \frac{1}{2}\delta^2$: the linear (MAE-like) component for large errors

The $-\frac{1}{2}\delta^2$ term ensures continuity of the loss and its gradient at the transition point $|\hat{y}_i - y_i| = \delta$. Huber loss is fully differentiable everywhere (unlike MAE), and it clips the gradient magnitude for outliers (unlike MSE). The hyperparameter $\delta$ controls where the transition happens. A common default is $\delta = 1.0$.

The gradient of Huber loss is:

$$\frac{\partial \mathcal{L}_{\delta}}{\partial \hat{y}_i} = \begin{cases} \hat{y}_i - y_i & \text{if } |\hat{y}_i - y_i| \leq \delta \\ \delta \cdot \text{sign}(\hat{y}_i - y_i) & \text{if } |\hat{y}_i - y_i| > \delta \end{cases}$$

For small errors, the gradient scales linearly with error (like MSE), giving strong correction signals proportional to mistake size. For large errors, the gradient is capped at $\pm\delta$, preventing outliers from dominating parameter updates. Huber loss is a principled way to get the best of both worlds: fast convergence near the optimum (quadratic behavior) with robustness to outliers (linear behavior).

In NLP, regression losses appear in tasks like predicting sentiment scores, estimating semantic similarity as a real-valued score, and quality estimation for machine translation. For any of these, Huber loss is a defensible default unless you have strong prior knowledge about the noise distribution.