Perplexity Evaluation › Calculating Perplexity: A Worked Example

Let us trace through a concrete example to see how perplexity accumulates across a sequence. Consider the sentence: "The cat sat."

Using a hypothetical subword tokenizer, this might become: ["The", "cat", "sat", "."]

Assume our model produces the following probabilities for each token given the previous context:

The average log probability is:

$$\begin{aligned} \text{Average log probability} &= \frac{-1.897 + (-2.526) + (-2.120) + (-1.386)}{4} \\ &= \frac{-7.929}{4} \\ &= -1.982 \end{aligned}$$

The perplexity is:

$$\exp(1.982) \approx 7.26$$

This means the model is about as uncertain as if it were choosing among 7.26 equally likely options at each position. Compare this to a uniform distribution over a vocabulary of 10,000 tokens, which would have perplexity 10,000. The model has reduced the uncertainty by a factor of roughly 1,378 compared to random guessing, showing it has learned meaningful patterns from the training data.

It is worth pausing on what these individual token probabilities reveal. The period at position 4 receives the highest probability (0.25) because punctuation at the end of a complete sentence is highly predictable. "The" at position 1 receives lower probability (0.15) because it must be predicted from nothing, and many different words could plausibly begin a sentence. "Cat" at position 2 is even harder to predict because while "the" is clearly a determiner requiring a noun, thousands of nouns are plausible. The model's probability for "sat" (0.12) reflects its learned association between cats and the action of sitting, informed by the full context "The cat."

Now consider what happens with a different tokenization. A more aggressive BPE merge might produce: ["The", "cat", "sat."]

If the model assigns probabilities 0.15, 0.08, and 0.10 respectively:

$$\begin{aligned} \text{Average log probability} &= \frac{-1.897 + (-2.526) + (-2.303)}{3} \\ &= -2.242 \end{aligned}$$ $$\text{Perplexity} = \exp(2.242) \approx 9.41$$

Notice that with fewer tokens, the perplexity is higher even though the model's predictions are similar. This illustrates why comparing raw perplexities across tokenization schemes is misleading. The second tokenization combines the word "sat" with the period, creating a compound token that is rarer and harder to predict. Even if the model assigns reasonable probability to this combination, the higher perplexity results from having fewer tokens to average over and the increased difficulty of predicting compound tokens.