Cross-Entropy Loss › Cross-Entropy and Perplexity

As established in our discussion of Perplexity, these two metrics are intimately related through a simple mathematical transformation. Understanding this relationship helps you interpret evaluation results and compare models across different papers and frameworks, as they measure the same underlying quantity but present it in different units that emphasize different aspects of model performance.

Perplexity is defined as the exponential of cross-entropy:

$$\begin{aligned} \text{Perplexity} &= \exp(H(P, Q)) \\ &= \exp\left(-\frac{1}{N} \sum_{i=1}^{N} \log P(x_i)\right) \end{aligned}$$

where:

• $\text{Perplexity}$: the effective branching factor or exponential of the cross-entropy
• $H(P, Q)$: the cross-entropy between true and predicted distributions
• $N$: the total number of tokens in the evaluation set
• $P(x_i)$: the model's predicted probability of the $i$-th true token $x_i$
• $\exp$: the exponential function (using the same base as the logarithm in cross-entropy)

The exponential uses the same base as the logarithm in the cross-entropy calculation. In natural language processing:

• When using natural logarithm ($\ln$) for cross-entropy, perplexity uses $e$ as the base
• When using base-2 logarithm ($\log_2$) for cross-entropy, perplexity uses 2 as the base

Both formulations measure the same underlying quantity, just on different scales. Cross-entropy in nats (natural log) or bits ($\log_2$) provides an additive measure of uncertainty, while perplexity provides a multiplicative measure that can be interpreted as the effective branching factor at each prediction step. If your model has a perplexity of 100, it is as uncertain as if it were choosing uniformly among 100 equally likely options at each step.

Cross-entropy offers advantages during training and optimization:

• Differentiability: Cross-entropy is directly differentiable and is the loss function for gradient computation. We can backpropagate through the cross-entropy calculation to update model weights.
• Additivity: Cross-entropy losses add across tokens, making aggregation straightforward. We can sum losses across batches and sequences without complications.
• Numerical stability: Working in log space avoids underflow when multiplying many small probabilities. Summing logs is numerically safer than multiplying probabilities directly.

Perplexity offers advantages for human interpretation:

• Scale intuition: A perplexity of 100 is more meaningful than a cross-entropy of 4.6 nats. Humans can intuitively grasp the difference between perplexity 50 and 100 better than between 3.9 and 4.6 nats.
• Vocabulary-relative performance: Perplexity naturally scales with vocabulary size. This allows rough comparison across models with different vocabularies. A perplexity equal to vocabulary size indicates random guessing. This provides a clear baseline.
• Historical convention: Perplexity remains the standard reporting metric for language model benchmarks. Research papers and leaderboards universally report perplexity, which makes it necessary for comparing with prior work.

In practice, modern frameworks often report both: cross-entropy for training curves and model selection, perplexity for final evaluation and paper results. During development, you might monitor cross-entropy to catch training instabilities early, then report perplexity in your final results to communicate with the broader research community.