Cross-Entropy Loss › Worked Example

Let's walk through a concrete calculation to solidify these concepts. Consider a tiny language model with a vocabulary of four tokens: {the, cat, sat, mat}. The model predicts the next token given the context "the cat". This simplified scenario allows us to trace through the calculations manually and verify that our mathematical understanding matches our computational implementation.

Suppose our model outputs the following probabilities for the next token:

The true next token is sat.

The true next token is sat. The cross-entropy loss is simply:

$$\begin{aligned} H &= -\log(0.7) \\ &\approx 0.357 \text{ nats} \end{aligned}$$

where:

• $H$: the cross-entropy loss for this single prediction
• $0.7$: the model's predicted probability for the true token (sat)

Converting to bits (dividing by $\ln 2$):

$$\begin{aligned} H_{\text{bits}} &= -\log_2(0.7) \\ &\approx 0.515 \text{ bits} \end{aligned}$$

where:

• $H_{\text{bits}}$: the cross-entropy measured in bits
• $0.7$: the model's predicted probability for the true token

The perplexity is:

$$\begin{aligned} \text{Perplexity} &= \exp(0.357) \\ &\approx 1.43 \end{aligned}$$

where:

• $\exp$: the exponential function with base $e$
• $0.357$: the cross-entropy in nats from the previous calculation

Or in base-2:

$$\begin{aligned} \text{Perplexity} &= 2^{0.515} \\ &\approx 1.43 \end{aligned}$$

where:

• $2$: the base of the logarithm (corresponding to bits)
• $0.515$: the cross-entropy in bits from the previous calculation

This low perplexity indicates the model is quite confident and correct. If the model had predicted uniform probabilities (0.25 each):

$$\begin{aligned} H &= -\log(0.25) \\ &= \ln 4 \\ &\approx 1.386 \text{ nats} \end{aligned}$$

where:

• $H$: the cross-entropy loss for uniform prediction
• $0.25$: the uniform probability for each of the 4 tokens

$$\text{Perplexity} = 4$$

where:

• $4$: the calculated perplexity value, equal to the vocabulary size for uniform distribution

This matches our intuition: uniform random guessing among 4 items yields perplexity 4. The model with 0.7 probability on the correct answer is effectively as uncertain as if it were choosing among only 1.43 equally likely options. This shows its high confidence.

Now consider a complete sentence: "the cat sat mat" (we omit the final punctuation for simplicity). The model processes this autoregressively, conditioning each prediction on all previous tokens:

1. Context: <bos>, True: the, Prob: 0.4, Loss: $-\ln(0.4) \approx 0.916$
2. Context: the, True: cat, Prob: 0.3, Loss: $-\ln(0.3) \approx 1.204$
3. Context: the cat, True: sat, Prob: 0.7, Loss: $-\ln(0.7) \approx 0.357$
4. Context: the cat sat, True: mat, Prob: 0.5, Loss: $-\ln(0.5) \approx 0.693$

The average cross-entropy is:

$$\begin{aligned} \mathcal{L} &= \frac{0.916 + 1.204 + 0.357 + 0.693}{4} \\ &\approx 0.793 \text{ nats} \end{aligned}$$

where:

• $\mathcal{L}$: the average cross-entropy loss over the sequence
• $0.916, 1.204, 0.357, 0.693$: the individual token losses (negative log probabilities) for each position
• $4$: the number of tokens in the sequence

Notice how token 2 ("cat") has the highest individual loss despite a moderate probability of 0.3. This reflects higher uncertainty at that position: many words could follow "the" in English. Token 3 ("sat") benefits from richer context and receives a much better prediction. This position-by-position view illustrates how the context window gradually reduces uncertainty as the model accumulates information.

The perplexity is:

$$\begin{aligned} \text{Perplexity} &= \exp(0.793) \\ &\approx 2.21 \end{aligned}$$

where:

• $\exp$: the exponential function with base $e$
• $0.793$: the average cross-entropy in nats

If the text contains 15 characters (including spaces), the bits per character would be:

$$\begin{aligned} \text{BPC} &= \frac{0.793 \times 4 \text{ nats} \times 1.443 \text{ bits/nat}}{15 \text{ chars}} \\ &\approx 0.305 \text{ BPC} \end{aligned}$$

where:

• $\text{BPC}$: bits per character, the compression efficiency metric
• $0.793$: the average cross-entropy loss in nats per token
• $4$: the number of tokens in the sequence
• $1.443$: the conversion factor from nats to bits ($1/\ln 2$)
• $15$: the total number of characters in the original text

Note the conversion factor: 1 nat = $\frac{1}{\ln 2} \approx 1.443$ bits. This conversion is necessary because our initial cross-entropy was computed using natural logarithms, but BPC is traditionally reported in bits.