Cross-Entropy Loss › Cross-Entropy Loss

Language models learn by predicting the next token in a sequence. But how do we quantify the gap between a model's predictions and reality? We need a mathematical measure that can compare the probability distribution output by our model against the actual observed data. This provides a single scalar value that indicates how well our predictions align with truth. Cross-entropy loss provides the mathematical foundation for this measurement, serving as both the training objective that guides gradient descent through the parameter space and the evaluation metric that tells us how well our model understands the statistical patterns of language.

As we explored in our discussion of Perplexity, language models output probability distributions over possible next tokens. At each position in a sequence, the model produces a vector of probabilities representing its confidence that each vocabulary item will appear next. Cross-entropy gives us a principled way to measure how far these predicted distributions diverge from the true distributions we observe in training data. It connects information theory, probability, and optimization into a single coherent framework that drives modern language AI. This framework allows us to transform the abstract problem of learning language patterns into a concrete optimization task where we can compute gradients and update model weights systematically.

Unlike perplexity, which expresses model uncertainty as an intuitive "effective vocabulary size," cross-entropy operates in the domain of information theory, measuring the average number of bits needed to encode the true data using a code optimized for the model's predicted distribution. This perspective reveals basic limits on compression and provides the optimization target for virtually all modern language models, from the causal objectives discussed in Causal Language Modeling to the masked objectives in Masked Language Modeling. Understanding cross-entropy provides deep insight into why language models work, how they improve during training, and what basic limits constrain their performance.

One reason cross-entropy occupies such a central position in language model training is that it connects naturally to maximum likelihood estimation, the workhorse of classical statistics. When you minimize cross-entropy loss over a dataset, you are simultaneously maximizing the probability that the model assigns to the observed training tokens. This equivalence means cross-entropy is not an arbitrary choice of loss function imposed by convention. It is the theoretically principled answer to the question of how to measure the quality of a probabilistic model. Later in this chapter, we will make this connection precise and see why it justifies using cross-entropy as the objective for training neural language models of any size.