Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Perplexity › From Cross-Entropy to Perplexity

Michael BrenndoerferMarch 26, 20255 min read

Language AI HandbookAll chapters

Part 2 of 67Classical Text Representations

Chapter 10Perplexity › From Cross-Entropy to Perplexity

Reading section 4 of 17 · From Cross-Entropy to Perplexity · 5 minSection 4 of 17 · 5 min

Section 4 of 17 · 5 min read

Perplexity › From Cross-Entropy to PerplexityOne section · Sections in this chapter4 / 17 ↓

Sections in this chapter17

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Cross-entropy is useful, but "2.7 bits per word" doesn't immediately convey how good a model is in intuitive terms. Is that good? Bad? It depends on the vocabulary size and the inherent predictability of the text. Comparing cross-entropy values across different models or tasks requires converting the number back into something more interpretable.

Perplexity solves this by converting bits back to a count, specifically the number of equally likely choices the model faces at each step. The transformation is simple:

\text{PP}(W) = 2^{H(P, Q)}

where:

$\text{PP}(W)$ : perplexity of the model on word sequence $W$
$H(P, Q)$ : cross-entropy between the true distribution $P$ and model distribution $Q$

Why raise 2 to the power of cross-entropy? Because entropy measures bits, and each bit represents a binary choice that doubles the number of possibilities. A cross-entropy of 3 bits means $2^3 = 8$ equally likely choices; 10 bits means $2^{10} = 1024$ choices. By exponentiating, we convert from the abstract "bits" unit back to the concrete "number of alternatives" unit.

Expanding this using our practical cross-entropy formula gives us the working definition of perplexity:

\text{PP}(W) = 2^{-\frac{1}{N} \sum_{i=1}^{N} \log_2 Q(w_i)}

where:

$N$ : total number of words in the test sequence
$w_i$ : the $i$ -th word in the sequence
$Q(w_i)$ : model's probability for word $w_i$ , given its context

Using properties of logarithms, we can derive an equivalent product form that makes the geometric mean interpretation explicit. The derivation proceeds as follows:

Step 1: Start with the exponential form:

\text{PP}(W) = 2^{-\frac{1}{N} \sum_{i=1}^{N} \log_2 Q(w_i)}

Step 2: Move the $-\frac{1}{N}$ inside using $2^{ab} = (2^a)^b$ :

\text{PP}(W) = \left( 2^{\sum_{i=1}^{N} \log_2 Q(w_i)} \right)^{-\frac{1}{N}}

Step 3: Apply $2^{\sum_i \log_2 x_i} = \prod_i x_i$ (the sum of logs equals the log of a product):

\text{PP}(W) = \left( \prod_{i=1}^{N} Q(w_i) \right)^{-\frac{1}{N}}

This shows that perplexity is the geometric mean of the inverse probabilities. Equivalently:

\text{PP}(W) = \frac{1}{\sqrt[N]{\prod_{i=1}^{N} Q(w_i)}}

where $\sqrt[N]{\cdot}$ denotes the $N$ -th root.

The geometric mean matters here for two important reasons. First, unlike the arithmetic mean, the geometric mean is sensitive to very small probabilities. A single word with near-zero probability will dramatically increase perplexity. If any $Q(w_i) \approx 0$ , then $\prod_i Q(w_i) \approx 0$ , making $\text{PP} \to \infty$ . This is exactly what we want: a good language model should not assign tiny probabilities to words that occur in real text. Second, the geometric mean makes perplexity scale-independent in a useful way: what matters is the relative confidence of predictions, not the absolute probability values.

Perplexity

The perplexity of a language model on a test set is the inverse probability of the test set, normalized by the number of words. It can be interpreted as the weighted average number of choices the model faces at each step. Lower perplexity means the model is less "perplexed" by the test data, assigning higher probability to what occurs.

For language models that condition on context, like n-gram models, we use conditional probabilities. The formula above assumed each word probability $Q(w_i)$ was independent, but real language models predict each word based on its history. The proper perplexity formula for such models is:

\text{PP}(W) = \sqrt[N]{\prod_{i=1}^{N} \frac{1}{P(w_i \mid w_1, \ldots, w_{i-1})}}

where:

$W = w_1, w_2, \ldots, w_N$ : the test sequence of $N$ words
$P(w_i \mid w_1, \ldots, w_{i-1})$ : the probability of word $w_i$ given all preceding words, the model's conditional prediction
$\sqrt[N]{\cdot}$ : the $N$ -th root (geometric mean)

In practice, we work with log probabilities to avoid numerical underflow. When multiplying many small probabilities (each less than 1), the product quickly approaches zero, causing floating-point arithmetic to lose precision entirely. By working in log space, we convert products to sums:

\text{PP}(W) = 2^{-\frac{1}{N} \sum_{i=1}^{N} \log_2 P(w_i \mid w_1, \ldots, w_{i-1})}

This is numerically stable because we're adding log probabilities (negative numbers typically in the range -3 to -15 for typical words) rather than multiplying tiny probabilities (numbers like $10^{-5}$ to $10^{-20}$ that quickly underflow to zero).

In[8]:

Code

def perplexity(test_tokens, model_log_prob):
    """
    Calculate perplexity of a model on test data.

    Args:
        test_tokens: List of tokens from test corpus
        model_log_prob: Function returning log2 P(token | context)
    """
    log_prob_sum = 0.0
    count = 0

    for i, token in enumerate(test_tokens):
        log_p = model_log_prob(token, i, test_tokens)
        if log_p == float("-inf"):
            return float("inf")  # Zero probability encountered
        log_prob_sum += log_p
        count += 1

    cross_ent = -log_prob_sum / count
    return 2**cross_ent


def unigram_log_prob(token, position, tokens):
    """Return log2 P(token) for unigram model with smoothing."""
    count = word_counts.get(token, 0)
    vocab_size = len(word_counts) + 1
    prob = (count + 1) / (total_words + vocab_size)
    return np.log2(prob)

Out[9]:

Console

Test corpus: the cat sat on the rug
Perplexity: 7.31
Effective branching factor: ~7 choices per word

Now the interpretation is immediate: a perplexity around 6-7 means the model faces roughly the same uncertainty as choosing uniformly among 6-7 equally likely words at each position. For a simple unigram model on a small corpus, this is reasonable. The model has learned that some words (like "the") are much more common than others, so it's not completely in the dark about what comes next.