Cross-Entropy Loss › Information Theory Foundations

To understand cross-entropy, we must first grasp entropy and how it quantifies uncertainty. The concept of entropy originates from thermodynamics and statistical mechanics, but Claude Shannon adapted it for information theory in 1948. This provides the mathematical vocabulary for measuring information content. Shannon's framework allows us to quantify how much information is contained in a random event, which directly translates to how surprised we should be when observing different outcomes. For language modeling, this means measuring how unexpected each token is given our current understanding of the language's statistical structure.

Shannon's key insight was that information and uncertainty are two sides of the same coin. An event carries more information precisely because it was unexpected. When a coin lands heads after you knew it would land heads because it is double-headed, you learn nothing. When that same coin lands heads despite being weighted toward tails, you learn something meaningful. This seemingly philosophical distinction has concrete mathematical consequences that turn out to be exactly what we need to train language models.

Entropy measures the uncertainty inherent in a probability distribution. It answers the question: on average, how surprised will we be when we observe an outcome from this distribution? For a discrete distribution $P$ over outcomes $x$, entropy is defined as:

$$H(P) = -\sum_{x} P(x) \log P(x)$$

where:

• $H(P)$: the entropy of probability distribution $P$, measured in bits (base-2 log) or nats (natural log)
• $x$: a possible outcome from the set of all events
• $P(x)$: the probability of outcome $x$ occurring
• $\log$: the logarithm function (typically base 2 for bits or natural log for nats)

The logarithm ensures that rare events contribute more to the total uncertainty than common ones. This mathematical property aligns with our intuition: when an unlikely event occurs, we experience more surprise and receive more information than when a common event occurs. When an event is certain ($P(x) = 1$), its contribution is zero because we gain no new information from observing something we already knew would happen. When an event is impossible ($P(x) = 0$), we define $0 \cdot \log 0 = 0$ by continuity. This reflects that impossible events contribute nothing to the entropy since they never occur.

To see why the logarithm specifically encodes surprise, consider two independent events each with probability $p$. The probability of both occurring together is $p^2$. If surprise is additive (seeing two independent rare events is twice as surprising as seeing one), then the measure of surprise should satisfy $f(p^2) = 2f(p)$. The only continuous function satisfying this property for all probabilities is the logarithm. So $-\log P(x)$ is the only mathematically consistent measure of surprise for an event with probability $P(x)$, and entropy is the expected value of this surprise over all possible outcomes.

Consider a vocabulary of 50,000 tokens. A uniform distribution where every token has probability $1/50,000$ yields entropy:

$$\begin{aligned} H &= -\sum_{i=1}^{50000} \frac{1}{50000} \log_2 \frac{1}{50000} \\ &= \log_2 50000 \\ &\approx 15.6 \text{ bits} \end{aligned}$$

This means we need approximately 15.6 bits on average to encode each token if we use an optimal code designed for this uniform distribution. This represents the worst-case scenario for compression, where no token is more predictable than any other. In contrast, if our model is perfect and assigns probability 1 to the correct token and 0 elsewhere, entropy drops to zero, requiring no bits to transmit the message because there is no uncertainty. The difference between these extremes represents the potential for compression that good language models exploit.

Real language is far from uniform. Common words like "the" and "is" appear far more frequently than rare technical terms. Good models learn these frequency differences and assign higher probability to common tokens, reducing average surprise below the maximum. The entropy of real English text has been estimated at 0.6 to 1.2 bits per character depending on context length, far below the 4.7 bits per character that would be required for a uniform distribution over the 26 letters and a space character. This gap is what enables compression algorithms, and it is the gap that language models are trained to exploit.

While entropy measures uncertainty within a single distribution, the Kullback-Leibler (KL) divergence measures how one probability distribution $Q$ diverges from a reference distribution $P$. It quantifies the inefficiency of assuming that the distribution is $Q$ when the true distribution is $P$:

$$D_{KL}(P \| Q) = \sum_{x} P(x) \log \frac{P(x)}{Q(x)}$$

where:

• $D_{KL}(P \| Q)$: the Kullback-Leibler divergence from distribution $Q$ to reference distribution $P$
• $P(x)$: the probability of outcome $x$ under the true distribution $P$
• $Q(x)$: the probability of outcome $x$ under the model distribution $Q$
• $x$: an outcome from the support of $P$

KL divergence is always non-negative and equals zero if and only if $P = Q$ everywhere. This non-negativity follows from Jensen's inequality applied to the concave logarithm function. It represents the extra bits required when using a code optimized for $Q$ to encode data that follows distribution $P$. This interpretation reveals that KL divergence measures the penalty we pay for using the wrong model: if we build a compression scheme based on our predicted distribution $Q$, but reality follows $P$, we will waste bits proportional to the KL divergence.

The asymmetry of KL divergence is worth understanding carefully. $D_{KL}(P \| Q)$ is not the same as $D_{KL}(Q \| P)$ in general. When we minimize $D_{KL}(P \| Q)$ with respect to $Q$, we penalize cases where $P(x) > 0$ but $Q(x) \approx 0$. This is called the "forward" or "inclusive" KL, and it encourages $Q$ to cover all regions where $P$ has mass. When we minimize $D_{KL}(Q \| P)$ (the "reverse" or "exclusive" KL), we penalize cases where $Q(x) > 0$ but $P(x) \approx 0$, which encourages $Q$ to concentrate on modes of $P$. Language model training uses the forward KL form through cross-entropy, which means models are penalized for assigning low probability to tokens that occur in training data, encouraging broad coverage of plausible continuations.

However, KL divergence has an asymmetry that matters for machine learning: it requires knowing the true distribution $P$ over all possible outcomes. In practice, we only observe specific outcomes from $P$, not the full distribution. We see individual tokens in our training data, but we do not know the complete probability distribution over all possible next tokens for every context. This limitation prevents us from computing the KL divergence directly during training, necessitating the use of cross-entropy instead.