Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

N-gram Language Models › Evaluating Language Models: Perplexity

Michael BrenndoerferMarch 23, 20258 min read

Language AI HandbookAll chapters

Part 2 of 67Classical Text Representations

Chapter 8N-gram Language Models › Evaluating Language Models: Perplexity

Reading section 7 of 15 · Evaluating Language Models: Perplexity · 8 minSection 7 of 15 · 8 min

Section 7 of 15 · 8 min read

N-gram Language Models › Evaluating Language Models: PerplexityOne section · Sections in this chapter7 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

So far we have talked about probability as the goal of language modeling, but how do we measure whether one language model is better than another? We need an evaluation metric that captures how well a model assigns probability to new, held-out text.

The most widely used evaluation metric for language models is perplexity. Perplexity measures how "surprised" a model is by new text. A model that assigns high probability to the test data is not very surprised, giving low perplexity. A model that assigns low probability to the test data is very surprised, giving high perplexity.

Defining PerplexityLink Copied

Perplexity is defined as the inverse probability of the test set, normalized by the number of words. For a test sequence $W = w_1, w_2, \ldots, w_N$ of $N$ words, perplexity is:

\text{PP}(W) = P(w_1, w_2, \ldots, w_N)^{-1/N}

where:

$P(w_1, w_2, \ldots, w_N)$ : the probability assigned to the test sequence by the model
$-1/N$ : the exponent normalizes for sequence length, so longer sequences are comparable to shorter ones
The outer operation takes the $N$ -th root (or equivalently, raises to the power $-1/N$ )

Expanding the sequence probability using the chain rule, perplexity becomes:

\text{PP}(W) = \left(\prod_{i=1}^{N} P(w_i|w_1, \ldots, w_{i-1})\right)^{-1/N}

Using the n-gram Markov approximation for a bigram model:

\text{PP}(W) = \left(\prod_{i=1}^{N} P(w_i|w_{i-1})\right)^{-1/N}

This is equivalent to the geometric mean of the inverse bigram probabilities. Minimizing perplexity is equivalent to maximizing the log probability of the test set per word.

In practice, we compute perplexity in log space to avoid numerical underflow:

\begin{aligned} \log \text{PP}(W) &= -\frac{1}{N} \log P(W) \\ &= -\frac{1}{N} \sum_{i=1}^{N} \log P(w_i|w_{i-1}) \end{aligned}

and then exponentiate:

\text{PP}(W) = \exp\!\left(-\frac{1}{N} \sum_{i=1}^{N} \log P(w_i|w_{i-1})\right)

where:

$N$ : number of words in the test set
$\log P(w_i|w_{i-1})$ : log probability of each word given its context (a negative number)
The average negative log probability gives the per-word entropy in nats
Exponentiating converts entropy back to a perplexity value

Interpreting PerplexityLink Copied

Perplexity has an intuitive interpretation as the effective branching factor of the language model. If every word has exactly the same number of equally likely continuations, perplexity equals that branching factor. A perplexity of 100 means the model is as confused as if it were choosing uniformly among 100 equally likely options at each step.

Consider the extremes:

Perplexity = 1: The model is perfectly certain about every word. Given the context, only one continuation is possible. This happens when text is entirely predictable (like a model evaluated on its own training data with no unseen n-grams).
Perplexity = V (vocabulary size): The model is completely uncertain, equivalent to a uniform distribution over all vocabulary words. This happens when context provides no information.
Perplexity in between: Real language models fall somewhere between these extremes. A good bigram model on English text might achieve a perplexity of 50 to 200, depending on the corpus and vocabulary.

Lower perplexity is always better, but the absolute value matters less than relative comparisons between models evaluated on the same test set.

In[19]:

Code

def perplexity(model, test_sentences):
    """Calculate perplexity of a language model on test sentences."""
    total_log_prob = 0.0
    total_words = 0

    for sentence in test_sentences:
        toks = sentence.lower().split()
        n = model.n
        padded = ["<s>"] * (n - 1) + toks + ["</s>"]

        for i in range(n - 1, len(padded)):
            context = tuple(padded[i - n + 1 : i])
            word = padded[i]
            prob = model.probability(word, context)
            if prob <= 0:
                # If any word has zero probability, perplexity is undefined
                return float("inf")
            total_log_prob += math.log(prob)
            total_words += 1

    # Perplexity = exp(-average log probability)
    avg_log_prob = total_log_prob / total_words
    return math.exp(-avg_log_prob)

Building a Complete N-gram Language Model ClassLink Copied

Let us build a reusable n-gram language model class that encapsulates everything we have covered so far:

In[20]:

Code

class NgramLanguageModel:
    """An n-gram language model with MLE estimation."""

    def __init__(self, n=2):
        self.n = n
        self.ngram_counts = Counter()
        self.context_counts = Counter()
        self.vocab = set()

    def train(self, text):
        """Train the model on text."""
        sentences = text.strip().split("\n")

        for sentence in sentences:
            if not sentence.strip():
                continue

            toks = sentence.lower().split()
            toks = ["<s>"] * (self.n - 1) + toks + ["</s>"]

            self.vocab.update(toks)

            for i in range(len(toks) - self.n + 1):
                ngram = tuple(toks[i : i + self.n])
                context = tuple(toks[i : i + self.n - 1])

                self.ngram_counts[ngram] += 1
                self.context_counts[context] += 1

    def probability(self, word, context):
        """Calculate P(word|context) using MLE."""
        if isinstance(context, str):
            context = tuple(context.split())

        ngram = context + (word,)

        ngram_count = self.ngram_counts.get(ngram, 0)
        context_count = self.context_counts.get(context, 0)

        if context_count == 0:
            return 0.0

        return ngram_count / context_count

    def sentence_log_probability(self, sentence):
        """Calculate log probability of a sentence."""
        toks = sentence.lower().split()
        toks = ["<s>"] * (self.n - 1) + toks + ["</s>"]

        log_prob = 0.0
        for i in range(self.n - 1, len(toks)):
            context = tuple(toks[i - self.n + 1 : i])
            word = toks[i]

            prob = self.probability(word, context)
            if prob == 0:
                return float("-inf")

            log_prob += math.log(prob)

        return log_prob

The NgramLanguageModel class encapsulates n-gram counting, context tracking, and probability computation. The train() method processes text and builds count tables. The probability() method returns $P(\text{word}|\text{context})$ using MLE. The sentence_log_probability() method computes the log probability of an entire sentence.

Let us train models of different orders on a larger corpus and compare their perplexity:

In[21]:

Code

# Larger training corpus
training_corpus = """
the cat sat on the mat
the cat ate the fish
the dog sat on the mat
the dog chased the cat
the cat ran from the dog
the dog ate the food
the bird flew over the tree
the bird sat on the branch
the cat watched the bird
the dog watched the cat
"""

# Train unigram, bigram, and trigram models
unigram_model = NgramLanguageModel(n=1)
bigram_model = NgramLanguageModel(n=2)
trigram_model = NgramLanguageModel(n=3)

unigram_model.train(training_corpus)
bigram_model.train(training_corpus)
trigram_model.train(training_corpus)

# Compare on test sentences
test_sentences = [
    "the cat sat on the mat",
    "the dog chased the cat",
    "the bird flew over the tree",
    "the cat ate the bird",
]

model_comparison = []
for sentence in test_sentences:
    scores = {
        "unigram": unigram_model.sentence_log_probability(sentence),
        "bigram": bigram_model.sentence_log_probability(sentence),
        "trigram": trigram_model.sentence_log_probability(sentence),
    }
    model_comparison.append((sentence, scores))

Out[22]:

Console

Model Comparison (log probabilities):
========================================================================
Sentence                               Unigram     Bigram    Trigram
------------------------------------------------------------------------
the cat sat on the mat                  -16.24      -5.30      -3.11
the dog chased the cat                  -13.35      -5.30      -3.62
the bird flew over the tree             -19.83      -5.99      -2.71
the cat ate the bird                    -13.17      -5.99       -inf

In[23]:

Code

# Compute perplexity on training data (where no zero probabilities exist)
# We use training data here since our tiny corpus has many unseen n-grams even on test
train_sentences_list = [
    s for s in training_corpus.strip().split("\n") if s.strip()
]

pp_uni = perplexity(unigram_model, train_sentences_list)
pp_bi = perplexity(bigram_model, train_sentences_list)
pp_tri = perplexity(trigram_model, train_sentences_list)

Out[24]:

Console

Perplexity on Training Data:
----------------------------------------
  Unigram model:  10.95
  Bigram model:   2.41
  Trigram model:  1.74

Lower perplexity = better model fit.
Trigram achieves best fit by using more context.

The results reveal key differences between model orders. The unigram model assigns similar probabilities to all sentences because it ignores word order entirely. The bigram model shows more variation, penalizing sentences with rare or unseen word transitions. The trigram model provides the finest discrimination. On training data, perplexity decreases with increasing n-gram order because more context allows more precise predictions. On held-out data with a small training corpus, this pattern can reverse if higher-order n-grams are too sparse.

Perplexity and EntropyLink Copied

Perplexity has a deep connection to information theory. The entropy of a language model is:

H(W) = -\frac{1}{N} \sum_{i=1}^{N} \log_2 P(w_i|w_1, \ldots, w_{i-1})

where:

$H(W)$ : the per-word entropy in bits
$N$ : total number of words in the test set
$\log_2$ : logarithm base 2 converts to bits
The entropy measures the average number of bits needed to encode each word given perfect context

Perplexity is simply $2^{H(W)}$ when using log base 2, or $\exp(H(W))$ when using natural logarithms. A perplexity of 100 corresponds to an entropy of about 6.6 bits per word, meaning the model is about as uncertain as if it had to choose uniformly among 100 equally likely options at each step.

This connection to information theory has a practical interpretation: a model with perplexity $k$ requires approximately $\log_2(k)$ bits per word to encode text under that model. Lower perplexity means more compressible text, which means the model has captured more of the statistical regularities in the language.