Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Perplexity › Cross-Entropy: The Foundation

Michael BrenndoerferMarch 26, 202510 min read

Language AI HandbookAll chapters

Part 2 of 67Classical Text Representations

Chapter 10Perplexity › Cross-Entropy: The Foundation

Reading section 3 of 17 · Cross-Entropy: The Foundation · 10 minSection 3 of 17 · 10 min

Section 3 of 17 · 10 min read

Perplexity › Cross-Entropy: The FoundationOne section · Sections in this chapter3 / 17 ↓

Sections in this chapter17

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

To measure how well a language model predicts text, we need a principled way to quantify "prediction quality." This is where information theory provides the perfect framework. The key insight is that prediction and compression are two sides of the same coin: if you can predict what comes next, you can compress it efficiently. Cross-entropy formalizes this connection, and perplexity translates it into an intuitive scale.

Let's build up to perplexity step by step, starting with the most fundamental question: how do we measure uncertainty?

Entropy: Quantifying UncertaintyLink Copied

Imagine you're playing a word-guessing game. Your friend thinks of a word, and you have to guess it using only yes/no questions. How many questions do you need?

The answer depends entirely on how predictable the word is. If your friend always picks from {"cat", "dog"} with equal probability, you need exactly one question ("Is it cat?"). But if they pick from a thousand equally likely words, you need about 10 questions (since $2^{10} = 1024$ ). This number of questions is exactly what entropy measures: the minimum average number of yes/no questions needed to determine an outcome.

For a probability distribution $P$ over a vocabulary $V$ , entropy is:

H(P) = -\sum_{w \in V} P(w) \log_2 P(w)

where:

$H(P)$ : entropy of distribution $P$ , measured in bits when using $\log_2$
$P(w)$ : probability of word $w$
The sum runs over all words in the vocabulary $V$

The formula might look abstract, but it captures a clear intuition. The term $-\log_2 P(w)$ is called the surprisal or information content of word $w$ . It measures how many bits of information you gain by learning that $w$ occurred:

A very likely word (say $P(w) = 0.5$ ) has surprisal $-\log_2(0.5) = 1$ bit
An unlikely word (say $P(w) = 0.0625$ ) has surprisal $-\log_2(0.0625) = 4$ bits
A certain word ( $P(w) = 1$ ) has surprisal $-\log_2(1) = 0$ bits (no surprise at all)

Entropy is the expected surprisal: we weight each word's surprisal by how often it occurs, $P(w)$ , and sum up. Rare words contribute more bits because they're more surprising when they occur, while common words contribute fewer because they're expected. Entropy captures the average information content across all outcomes.

The connection to coding theory is tight. If you want to design the most efficient binary code for transmitting words from this distribution, the optimal code assigns $-\log_2 P(w)$ bits to word $w$ . Frequently occurring words get short codes (few bits), rare words get long codes (many bits). The entropy is the minimum average number of bits needed by any such scheme, a theoretical limit known as Shannon's source coding theorem. A language model that assigns better probabilities to common words and lower probabilities to rare ones is essentially learning a better compression scheme for language.

Let's see this in action with a simple two-word vocabulary:

In[3]:

Code

import numpy as np


def entropy(probs):
    """Calculate entropy of a probability distribution."""
    # Filter out zero probabilities to avoid log(0)
    probs = np.array([p for p in probs if p > 0])
    return -np.sum(probs * np.log2(probs))


# Different probability distributions over two words
distributions = [
    ([0.5, 0.5], "Equal probability"),
    ([0.9, 0.1], "Skewed (90/10)"),
    ([0.99, 0.01], "Very skewed (99/1)"),
    ([1.0, 0.0], "Certain"),
]

Out[4]:

Console

Entropy for different distributions over {cat, dog}:
--------------------------------------------------
Equal probability         H = 1.0000 bits
Skewed (90/10)            H = 0.4690 bits
Very skewed (99/1)        H = 0.0808 bits
Certain                   H = -0.0000 bits

Notice the pattern: when both words are equally likely, entropy is maximal at 1 bit. You need one yes/no question. As one word dominates, entropy drops because the outcome becomes more predictable. When the outcome is certain, entropy is zero: no questions needed, no uncertainty remains.

Out[5]:

Visualization

Line plot showing entropy on y-axis versus probability of first outcome on x-axis, forming an inverted U shape peaking at 0.5. — Entropy of a binary distribution as a function of probability skew. Maximum entropy of 1 bit occurs at equal probabilities (p = 0.5), where uncertainty is highest. As one outcome dominates (p near 0 or 1), entropy approaches zero because the result becomes predictable. The shaded region highlights the range of meaningful uncertainty.

This gives us our first key insight: entropy measures the inherent unpredictability of a distribution. A good language model should have low entropy on real text because it can accurately predict what comes next. When a language model says "after 'the sky is', the next word is very likely 'blue'," it's operating in a low-entropy regime for that context, with very high certainty about the next word. When it encounters an ambiguous context where almost any continuation is plausible, it's in a high-entropy regime.

Cross-Entropy: Measuring Model MismatchLink Copied

Entropy tells us about the true distribution of language, but we don't have direct access to that. We only have our model's predictions. This is where cross-entropy enters the picture.

Cross-entropy asks: "If the true distribution is $P$ , but we use model $Q$ to make predictions, how many bits do we need on average?" The answer is always at least as many as entropy itself, and usually more, because our model isn't perfect. We're encoding outcomes using a code designed for distribution $Q$ , but the outcomes come from distribution $P$ .

H(P, Q) = -\sum_{w \in V} P(w) \log_2 Q(w)

where:

$H(P, Q)$ : cross-entropy of $P$ relative to $Q$ , measured in bits
$P(w)$ : true probability of word $w$ , from the actual data distribution
$Q(w)$ : model's predicted probability of word $w$
$V$ : the vocabulary, the set of all possible words
The sum computes the expected number of bits, weighting each word's encoding cost $-\log_2 Q(w)$ by its true frequency $P(w)$

The key relationship is: $H(P, Q) \geq H(P)$ . Cross-entropy is always at least as large as entropy. This inequality holds because using the "wrong" distribution $Q$ for encoding is never better than using the true distribution $P$ . The gap between them, called KL divergence, measures exactly how much our model's predictions differ from reality.

Think of it this way:

Entropy $H(P)$ : the minimum bits needed with a perfect model
Cross-entropy $H(P, Q)$ : the bits needed with our actual model $Q$
KL divergence $H(P, Q) - H(P)$ : the "cost" of using an imperfect model

The KL divergence is sometimes called the "inefficiency" of the model. It represents the wasted bits that come from using $Q$ instead of $P$ to design your encoding scheme. A model that perfectly captures the true distribution of language would have zero KL divergence and achieve the theoretical entropy minimum.

Cross-entropy decomposition showing how model quality affects the KL divergence component. The entropy of 2.5 bits represents the inherent uncertainty in the data, the irreducible minimum even with a perfect model. Better models have smaller KL divergence, getting closer to that theoretical minimum.
Model Quality	Entropy H(P)	KL Divergence	Cross-Entropy
Perfect Model	2.5 bits	0.0 bits	2.5 bits
Good Model	2.5 bits	0.5 bits	3.0 bits
Average Model	2.5 bits	1.5 bits	4.0 bits
Poor Model	2.5 bits	3.0 bits	5.5 bits
Random Guessing	2.5 bits	5.0 bits	7.5 bits

In practice, we don't know the true distribution $P$ . Instead, we approximate it using the empirical distribution from our test data. If word $w$ appears $C(w)$ times in a test corpus of $N$ words, we estimate:

\hat{P}(w) = \frac{C(w)}{N}

where:

$\hat{P}(w)$ : empirical probability of word $w$ (the "hat" notation indicates an estimate, not the true value)
$C(w)$ : count of how many times word $w$ appears in the test corpus
$N$ : total number of words in the test corpus

When we substitute this empirical distribution into the cross-entropy formula, something elegant happens. Since each word $w_i$ in the corpus contributes equally to the empirical distribution (each occurrence contributes $1/N$ to the probability mass), the weighted sum simplifies to a plain average:

H(\hat{P}, Q) = -\frac{1}{N} \sum_{i=1}^{N} \log_2 Q(w_i)

where:

$H(\hat{P}, Q)$ : cross-entropy between the empirical distribution $\hat{P}$ and model $Q$
$N$ : total number of words in the test corpus
$w_i$ : the $i$ -th word in the test corpus
$Q(w_i)$ : the probability our model assigns to word $w_i$

In words: cross-entropy is the average negative log probability that our model assigns to each word in the test corpus. Lower is better because it means the model assigned higher probabilities to the words that appeared. A model that assigns probability 0.1 to every word contributes $-\log_2(0.1) \approx 3.32$ bits per word. A model that assigns probability 0.5 to every word contributes just 1 bit. The better the model, the closer its assigned probabilities are to 1.0 for words that occur, and the lower the cross-entropy.

Let's compute this for a simple unigram model:

In[6]:

Code

def cross_entropy(test_tokens, model_probs):
    """
    Calculate cross-entropy of a model on test data.

    Args:
        test_tokens: List of tokens from test corpus
        model_probs: Function that returns P(token | context)
    """
    log_prob_sum = 0.0
    count = 0

    for i, token in enumerate(test_tokens):
        prob = model_probs(token, i, test_tokens)
        if prob > 0:
            log_prob_sum += np.log2(prob)
            count += 1
        else:
            # Handle zero probability (model assigns impossible)
            return float("inf")

    return -log_prob_sum / count


# Simple unigram model for demonstration
train_corpus = """
the cat sat on the mat
the dog sat on the rug
the cat chased the dog
the dog chased the cat
""".lower().split()

from collections import Counter

word_counts = Counter(train_corpus)
total_words = len(train_corpus)


def unigram_prob(token, position, tokens):
    """Return unigram probability P(token)."""
    count = word_counts.get(token, 0)
    # Add-1 smoothing to handle unseen words
    vocab_size = len(word_counts) + 1  # +1 for unknown
    return (count + 1) / (total_words + vocab_size)


# Test on held-out data
test_corpus = "the cat sat on the rug".split()

Out[7]:

Console

Test corpus: the cat sat on the rug
Cross-entropy: 2.8692 bits per word

A cross-entropy around 2.7 bits per word indicates moderate predictability. For comparison, a uniform distribution over our vocabulary of about 8 unique words would give $\log_2(8) = 3$ bits. Our unigram model does slightly better because it learned that some words (like "the") are more common than others.

The cross-entropy tells us how many bits on average our model needs to encode each word. This number has a concrete interpretation: if we used our model's probabilities to design a compression scheme, each word would require about this many bits on average. The better the model, the fewer bits needed, and the better the compression.