Linear Classifiers › Maximum Likelihood Estimation and Cross-Entropy

The cross-entropy loss is not an arbitrary choice. It arises naturally from maximum likelihood estimation (MLE), a foundational principle in statistics. Understanding this connection grounds the training process in probability theory and explains why cross-entropy is the "correct" objective.

Suppose we model the conditional probability $P(y = 1 \mid \mathbf{x}) = \sigma(\mathbf{w} \cdot \mathbf{x} + b)$. Given a training dataset $\{(\mathbf{x}_1, y_1), \ldots, (\mathbf{x}_N, y_N)\}$ of independent examples, the likelihood of observing this dataset under our model is:

For binary labels, this becomes $\prod_{i=1}^{N} \hat{p}_i^{y_i} (1 - \hat{p}_i)^{1 - y_i}$. Taking the log (which turns the product into a sum and is monotonically increasing, so maximizing log-likelihood is equivalent to maximizing likelihood):

Minimizing the negative log-likelihood is exactly minimizing the average cross-entropy loss. Maximum likelihood estimation tells us that the cross-entropy loss is the principled objective for logistic regression: we are finding the parameters that make the observed training labels as likely as possible under our model. This connection is fundamental: it means that training logistic regression produces well-calibrated probability estimates, not just good rankings.

Well-calibrated probabilities are practically important. If your sentiment classifier says 90% probability of positive for a particular review, and it is well-calibrated, then roughly 90% of similar reviews with that predicted probability will indeed be positive. This means you can use the raw probability outputs as decision inputs, alongside the hard class predictions. Poor calibration, often seen in models trained with mean squared error or in overconfident deep networks, can lead to systematic errors in downstream systems that rely on probability values.