Perplexity Evaluation › The Information-Theoretic Foundation

Before deriving the perplexity formula, it helps to understand why information theory provides the right language for thinking about language model quality. A language model assigns probabilities to sequences, and we want to know how well those probabilities match reality. This is precisely the question that information theory was designed to answer.

Claude Shannon's foundational insight, published in 1948, was that information can be quantified. The more surprising an event is, the more information it carries. If someone tells you that the sun rose this morning, they convey almost no information because you already knew that with near certainty. If they tell you it snowed in July in Phoenix, they convey a great deal of information because that event was unlikely.

Shannon formalized this intuition with the concept of entropy. For a discrete random variable $X$ with probability distribution $P$, entropy measures the average information content per sample:

$$H(X) = -\sum_{x} P(x) \log P(x)$$

where:

• $H(X)$: the entropy of the distribution $P$, measured in bits (base-2 log) or nats (natural log)
• $x$: a possible outcome of the random variable $X$
• $P(x)$: the probability of outcome $x$
• $-\log P(x)$: the information content (surprise) of observing outcome $x$

Entropy is maximized when all outcomes are equally likely (maximum uncertainty) and minimized when one outcome is certain (no uncertainty). For a fair six-sided die, entropy is $\log_2 6 \approx 2.58$ bits. For a biased coin that lands heads 99% of the time, entropy is close to zero.

Shannon also showed that entropy has a direct operational meaning: it is the minimum average number of binary questions needed to identify an outcome, or equivalently, the minimum number of bits needed to encode samples from the distribution. This compression interpretation is why perplexity has a natural meaning in terms of prediction difficulty.

In practice, we do not know the true distribution of language $P$. We have a model $Q$ (our language model) that approximates $P$. Cross-entropy measures the cost of encoding samples from $P$ using a code optimized for $Q$:

$$H(P, Q) = -\sum_{x} P(x) \log Q(x)$$

where:

• $H(P, Q)$: the cross-entropy between true distribution $P$ and model distribution $Q$
• $P(x)$: the true probability of outcome $x$ under the actual data distribution
• $Q(x)$: the probability assigned to $x$ by our model
• $-\log Q(x)$: the code length assigned to $x$ by our model

Cross-entropy is always at least as large as entropy: $H(P, Q) \geq H(P)$, with equality only when $Q = P$. The gap between them is the Kullback-Leibler (KL) divergence, which measures how much additional information we waste by using the wrong distribution. Language model training minimizes cross-entropy precisely because minimizing cross-entropy is equivalent to maximizing the likelihood of the training data under the model.

In language modeling, we observe a sequence of tokens $w_1, w_2, \ldots, w_N$ drawn from the true distribution $P$. Our model $Q$ assigns probabilities to these tokens. The empirical cross-entropy of the sequence under our model is:

$$H(P, Q) = -\frac{1}{N} \sum_{i=1}^{N} \log Q(w_i | w_1, \ldots, w_{i-1})$$

where:

• $N$: the total number of tokens in the evaluation sequence
• $w_i$: the $i$-th token in the sequence
• $Q(w_i | w_1, \ldots, w_{i-1})$: the probability our model assigns to $w_i$ given the preceding context

The $\frac{1}{N}$ normalization converts total log-likelihood into a per-token average, making the metric independent of sequence length. Without this normalization, longer sequences would always have lower total likelihood simply because we multiply more probabilities together.

Training language models by minimizing cross-entropy is equivalent to minimizing the KL divergence from the model distribution to the true data distribution. The KL divergence is:

$$D_{\text{KL}}(P \| Q) = H(P, Q) - H(P)$$

where:

• $D_{\text{KL}}(P \| Q)$: the KL divergence from model $Q$ to true distribution $P$
• $H(P, Q)$: the cross-entropy between $P$ and $Q$
• $H(P)$: the entropy of the true distribution $P$

Since $H(P)$ is a constant determined by the true data distribution and not by our model, minimizing cross-entropy and minimizing KL divergence are equivalent from an optimization perspective. When we report perplexity, we are indirectly reporting how close our model distribution is to the true data distribution, as measured by KL divergence plus the irreducible entropy of natural language.

This perspective clarifies what perplexity improvement means in practice. Each bit of improvement in cross-entropy corresponds to halving the perplexity (since $\text{PPL} = 2^{H}$ in bits). Moving from perplexity 100 to perplexity 50 means the model has reduced its average uncertainty by one bit per token, which in turn means it has approximately doubled its confidence in the most likely token at each step.