Perplexity Evaluation › The Mathematics of Perplexity

Perplexity is fundamentally about uncertainty. When we say a model has a perplexity of 100, we mean that at each position, the model is as uncertain as if it were choosing uniformly among 100 equally likely options. This interpretation connects directly to the concept of entropy from information theory. This provides an intuitive scale for measuring predictive performance. Rather than working with logarithmic loss values that can be difficult to interpret, perplexity gives us a linear scale where higher numbers indicate greater uncertainty and lower numbers indicate more confident, accurate predictions.

Recall from our discussion of language model training that we optimize models using cross-entropy loss. For a sequence of tokens $w_1, w_2, \ldots, w_N$, the cross-entropy measures the average number of bits needed to encode the sequence using the model's predicted distribution:

where:

• $H(W)$: the cross-entropy of the sequence $W$ (measured in bits if using base-2 logarithms, or nats if using natural logarithms)
• $N$: the total number of tokens in the sequence
• $w_i$: the $i$-th token in the sequence
• $w_1 \ldots w_{i-1}$: the context of all previous tokens before position $i$
• $P(w_i | w_1 \ldots w_{i-1})$: the conditional probability assigned to token $w_i$ by the model given the preceding context

This formula tells us how efficiently we can compress the text using the model's predictions. If the model assigns high probability to the actual next token, the logarithm of that probability will be close to zero (less negative), resulting in lower cross-entropy. Conversely, if the model is surprised by the actual token and assigns it low probability, the logarithm becomes a large negative number, increasing the cross-entropy.

Perplexity is simply the exponentiation of this cross-entropy:

where:

• $\text{Perplexity}(W)$: the perplexity of sequence $W$, representing the effective number of equally likely choices at each position
• $H(W)$: the cross-entropy of the sequence (as defined above)
• $2^{H(W)}$: the exponentiation using base 2, used when cross-entropy is measured in bits
• $\exp(\cdot)$: the exponential function using base $e$, used when cross-entropy is measured in nats (natural log)

We exponentiate the cross-entropy to return to a scale that represents the effective vocabulary size. While cross-entropy gives us the average number of bits needed per token, perplexity tells us how many equally likely options that corresponds to. This makes perplexity more interpretable: a perplexity of 100 immediately conveys that the model faces uncertainty equivalent to choosing among 100 equally probable outcomes, whereas a cross-entropy of 6.64 bits requires calculation to understand the scale of uncertainty.

The base of the exponent depends on the logarithm used. In deep learning frameworks, we typically use natural logarithms. This makes the formula $\exp(\text{cross-entropy})$. In information theory contexts using base-2 logarithms, perplexity becomes $2^{\text{cross-entropy}}$. The numerical value differs by a constant factor, but the interpretation remains consistent. In practice, when comparing models trained using standard deep learning frameworks, we use the natural log version exclusively. This keeps fair comparison since the constant factor cancels out when computing ratios or differences.

Modern language models use different training objectives that require adjustments to how we compute perplexity. The way we calculate perplexity must respect the directionality and conditioning of the model's predictions. This ensures that we evaluate the model on positions where it makes predictions during training.

Causal Language Models (GPT-style autoregressive models) predict each token conditioned only on previous tokens. For these models, we calculate perplexity using the standard chain rule decomposition:

where:

• $P(w_1, w_2, \ldots, w_N)$: the joint probability of the entire sequence of $N$ tokens
• $\prod_{i=1}^{N}$: the product over all token positions from $1$ to $N$ (multiplying individual probabilities)
• $w_i$: the token at position $i$
• $w_{<i}$: shorthand for all tokens preceding position $i$ (i.e., $w_1, w_2, \ldots, w_{i-1}$)
• $P(w_i | w_{<i})$: the conditional probability of token $w_i$ given all previous tokens

The chain rule decomposition reflects the autoregressive nature of these models. They factor the joint probability of the sequence into a product of conditional probabilities, each depending only on what came before. This mirrors how we generate text: we choose the first word, then the second given the first, and so on.

The log-likelihood sums over all positions where the model makes predictions:

where:

• $\log \text{Likelihood}$: the total log-probability of the sequence (sum of individual log-probabilities)
• $\sum_{i=1}^{N}$: the sum over all token positions from $1$ to $N$
• $\log P(w_i | w_{<i})$: the natural logarithm of the conditional probability for token $w_i$

We use the logarithm to convert products into sums, which prevents numerical underflow when dealing with very small probabilities. It also makes the optimization landscape smoother and more tractable during training.

Masked Language Models (BERT-style) present a complication. During training, these models predict only the masked positions using bidirectional context. For evaluation, we have two options:

1. Pseudo-log-likelihood: Score each position individually by masking it and computing the probability, summing across all positions. This is computationally expensive ($O(N)$ forward passes).
2. Adjusted perplexity: Compute perplexity only on the masked positions during evaluation, similar to training.

The pseudo-log-likelihood approach provides a rigorous evaluation that treats every position equally, but it requires running the model forward $N$ times for a sequence of length $N$, which makes it impractical for large-scale evaluation. The second approach, evaluating only on masked positions, aligns with the training objective but may not give a complete picture of the model's ability to model the full sequence distribution.

Most implementations use the pseudo-log-likelihood approach for rigorous evaluation, though it is impractical for large-scale benchmarking. This creates a tension between theoretical rigor and computational feasibility. When comparing masked language models to causal language models, we must be especially careful about which evaluation protocol we use, as pseudo-log-likelihood gives bidirectional models an advantage by letting them to see future context when scoring each position.

Prefix Language Models (T5-style encoder-decoder models) compute perplexity on the target sequence conditioned on the source. The calculation mirrors causal LM perplexity but treats the source as context:

where:

• $x$: the source sequence (input to the encoder)
• $y$: the target sequence of length $M$ (output from the decoder)
• $y_j$: the $j$-th token in the target sequence
• $y_{<j}$: all target tokens preceding position $j$ (previously generated tokens)
• $M$: the number of tokens in the target sequence
• $P(y_j | x, y_{<j})$: the conditional probability of generating token $y_j$ given the source $x$ and previous target tokens

In this architecture, the encoder processes the entire source sequence to create a contextualized representation, while the decoder generates the target sequence autoregressively. Perplexity measures how well the decoder predicts the actual target tokens given both the source context and the previously generated target tokens. This is particularly relevant for machine translation and summarization tasks, where we want to measure how well the model predicts reference translations or summaries.

A important distinction in modern evaluation is between token-level and word-level perplexity. Neural language models operate on subword tokens (BPE, WordPiece, SentencePiece), while traditional n-gram models used word-level vocabularies. This difference creates a basic challenge for comparing modern neural models to classical baselines or even to each other when they use different tokenization schemes.

Token-level perplexity is what we directly compute from model outputs. It measures the effective branching factor at each prediction step based on the model's token-level probabilities:

where:

• $\text{PPL}_{\text{token}}$: the token-level perplexity, representing the effective number of equally likely token choices at each position
• $T$: the total number of tokens in the sequence
• $t_t$: the token at position $t$ in the sequence (the specific token being predicted)
• $t_{<t}$: all tokens preceding position $t$ (the context available for prediction)
• $P(t_t | t_{<t})$: the conditional probability assigned to token $t_t$ by the model given the previous tokens

However, comparing token-level perplexities across different tokenizers is problematic. A model with a smaller vocabulary (and thus fewer tokens per word) will have more prediction steps per word, giving it more opportunities to accumulate probability mass. Conversely, a model with a larger vocabulary that uses more aggressive subword merging will have fewer tokens per word, meaning each prediction carries more weight in the final average.

Word-level perplexity attempts to normalize this by aggregating token probabilities into word probabilities. For a word $w$ split into subtokens $t_1, \ldots, t_k$:

where:

• $P(w)$: the joint probability of the complete word $w$
• $k$: the number of subword tokens that comprise the word $w$
• $t_j$: the $j$-th subword token of the word
• $t_{<j}$: all subword tokens preceding $t_j$ within the word
• $\text{context}$: the broader sentence context outside the current word
• $\prod_{j=1}^{k}$: the product of probabilities for each subword token

Then word-level perplexity becomes:

where:

• $\text{PPL}_{\text{word}}$: the word-level perplexity (normalized by word count)
• $W$: the total number of words in the sequence (not tokens)
• $w_i$: the $i$-th word in the sequence (aggregated from its subword tokens)
• $P(w_i)$: the probability of word $w_i$ computed from its subword token probabilities

Computing word-level perplexity requires careful handling of tokenization boundaries. Special tokens like <s> or </s> must be excluded, and subword units must be correctly grouped. Different tokenizers handle whitespace and punctuation differently, making cross-model comparison difficult even at the word level. For instance, some tokenizers treat punctuation as separate tokens while others attach it to words, which affects how we aggregate probabilities and count words.

A third option is bits per character (BPC), which normalizes perplexity to the character level regardless of tokenization:

where:

• $\text{BPC}$: bits per character, the average number of bits needed to predict each character
• $N$: the number of tokens in the sequence
• $\log_2 P(w_i | w_{<i})$: the log base-2 probability of each token (in bits, not nats)
• $\text{total characters}$: the number of characters in the raw text before tokenization

BPC sidesteps tokenization entirely by measuring how many bits the model requires to predict each character of text. This makes it the most consistent cross-model comparison metric, but it requires converting token-level log-probabilities back to character-level costs, which adds implementation complexity. Most contemporary model comparisons use token-level perplexity within controlled settings (same tokenizer) rather than attempting cross-tokenizer normalization. BPC remains valuable when comparing models trained on different tokenization schemes or when evaluating on non-Latin scripts where tokenization strategies differ radically.