Cross-Entropy Loss › Defining Cross-Entropy

The connection between cross-entropy and maximum likelihood estimation (MLE) explains the statistical basis for using cross-entropy to train probabilistic language models. Given a dataset of $N$ observed tokens $\{x_1, x_2, \ldots, x_N\}$, MLE seeks the model parameters $\theta$ that maximize the probability of the observed data:

$$\hat{\theta} = \arg\max_{\theta} \prod_{i=1}^{N} P(x_i | \text{context}_i; \theta)$$

where:

• $\hat{\theta}$: the maximum likelihood estimate of the model parameters
• $P(x_i | \text{context}_i; \theta)$: the model's probability of the $i$-th observed token given its context
• $\prod$: the product over all tokens in the dataset

Because products of many small probabilities rapidly underflow to zero numerically, we take the logarithm of both sides. This converts the product into a sum, which is numerically stable:

$$\begin{aligned} \hat{\theta} &= \arg\max_{\theta} \sum_{i=1}^{N} \log P(x_i | \text{context}_i; \theta) \\ &= \arg\min_{\theta} -\frac{1}{N} \sum_{i=1}^{N} \log P(x_i | \text{context}_i; \theta) \end{aligned}$$

where:

• $\sum_{i=1}^{N} \log P(x_i | \text{context}_i; \theta)$: the log-likelihood of the observed tokens
• $-\frac{1}{N} \sum_{i=1}^{N} \log P(x_i | \text{context}_i; \theta)$: the negative log-likelihood, normalized by dataset size

The final expression is exactly the cross-entropy loss averaged over the training set. Minimizing cross-entropy is mathematically identical to performing maximum likelihood estimation. This means we have a strong statistical justification for using cross-entropy: it is the objective that, as $N$ grows large, causes our model to converge to the true underlying data distribution (or the closest approximation within our model family). Any other loss function would not have this convergence guarantee in the same form.

In autoregressive language modeling, we predict the next token $x_t$ given previous tokens $x_{<t}$. This sequential factorization allows us to model complex joint distributions over sequences by breaking them down into manageable conditional probabilities. The probability of a sequence factorizes as:

$$P(x_1, x_2, \ldots, x_T) = \prod_{t=1}^{T} P(x_t | x_{<t})$$

where:

• $P(x_1, x_2, \ldots, x_T)$: the joint probability of the entire sequence
• $P(x_t | x_{<t})$: the conditional probability of token $x_t$ given all preceding tokens $x_{<t}$

This factorization is exact: the chain rule of probability guarantees it without any independence assumptions. The cross-entropy loss for a sequence of length $T$ is:

$$\mathcal{L} = -\frac{1}{T} \sum_{t=1}^{T} \log P(x_t | x_{<t}; \theta)$$

where:

• $\mathcal{L}$: the average cross-entropy loss over the sequence
• $T$: the length of the target sequence (number of tokens)
• $x_t$: the token at position $t$ in the sequence
• $x_{<t}$: all tokens preceding position $t$ (the context)
• $\theta$: the model parameters
• $P(x_t | x_{<t}; \theta)$: the model's predicted probability of token $x_t$ given previous tokens

This formulation reveals why cross-entropy is called the log-loss: we sum the negative log probabilities of each token. Lower probability assigned to the correct token results in higher loss, with the loss approaching infinity as the predicted probability approaches zero. This creates strong gradients when the model is confidently wrong, pushing the parameters aggressively to correct major errors.

The averaging by $T$ (sequence length) ensures that longer sequences do not automatically incur higher loss simply because they contain more tokens. This normalization allows us to compare loss values across sequences of different lengths and to aggregate loss across batches during training. Without this normalization, a long document would always appear to be modeled worse than a short sentence, even if the model's per-token accuracy was identical.

A necessary practical point: in transformer training, the model processes an entire sequence in a single forward pass, computing predicted probability distributions at every position simultaneously. For a sequence of length $T$, this produces $T$ losses in parallel. The positions at the beginning of the sequence receive less context than positions near the end, so the model must predict early tokens with limited information. This variability in difficulty across positions is averaged away by the uniform weighting in the loss, which treats all positions equally. Some researchers have proposed weighting later positions more heavily, since they benefit from richer context and thus provide cleaner learning signal, but this remains an active area of investigation.