Cross-Entropy Loss › Summary

Cross-entropy loss provides the mathematical foundation for training and evaluating language models, connecting information theory with practical optimization. Key insights to remember:

The core definition: cross-entropy measures the average bits needed to encode data from distribution $P$ using a code optimized for distribution $Q$:

where:

• $H(P, Q)$: the cross-entropy between the true distribution $P$ and the predicted distribution $Q$
• $x$: an outcome from the set of possible events
• $P(x)$: the probability of outcome $x$ under the true distribution $P$
• $Q(x)$: the probability of outcome $x$ under the model distribution $Q$

The key connections and implications:

• Relationship to perplexity: Perplexity equals the exponential of cross-entropy ($\exp(H)$), converting additive information measures into multiplicative branching factors that are easier to interpret intuitively.
• Equivalence to MLE: Minimizing cross-entropy maximizes the likelihood of training data, which makes it equivalent to maximum likelihood estimation under the model's parameterization and giving it a strong statistical justification.
• Gradient behavior: The gradient of the combined softmax-cross-entropy with respect to logits simplifies to $\hat{y}_j - y_j$, a bounded and interpretable signal that is large when the model is wrong and small when it is right.
• Bits per character: Normalizing by character count rather than token count allows fair comparison across different tokenization strategies, with state-of-the-art models achieving approximately 0.7-0.9 BPC on English text.
• Loss curves: Training dynamics typically show rapid initial improvement followed by gradual convergence following power-law relationships with compute and data.
• Numerical stability: Real implementations use the log-sum-exp trick and operate on logits rather than probabilities to avoid overflow and underflow in floating-point computation.

Cross-entropy is the universal training signal for Causal Language Modeling and Masked Language Modeling, but remember that low cross-entropy does not guarantee high-quality generation, factual accuracy, or aligned behavior. In the following chapters, we'll explore complementary metrics like BLEU and ROUGE that better capture generation quality, as well as benchmarks that evaluate capabilities beyond next-token prediction. The limitations of cross-entropy are as important to understand as its strengths: knowing what the training objective cannot measure helps us design better evaluation strategies and recognize when a model's low perplexity score might not translate to practical value.