Perplexity Evaluation › Perplexity Evaluation

Language models are probability distributions over sequences. They assign a likelihood to every possible continuation of text, from the most natural to the completely nonsensical. But how do we quantify how "good" these probability assignments are? How do we know if one model's distribution is better than another's?

Perplexity answers this question by measuring how well a probability model predicts a sample. It captures the basic tension in language modeling: assigning high probability to what occurs, while maintaining a coherent distribution over what might occur. A model with low perplexity is "less surprised" by real text, meaning it anticipated the words that appeared. This metric connects the probability estimation objective to text prediction, letting us assess whether our models have captured patterns of human language.

The intuition is elegant. When you read a sentence, you are rarely surprised by the next word. You know grammatical constraints, semantic plausibility, and the topic at hand. A language model that has learned these same patterns will assign high probability to likely continuations, and its perplexity will reflect this confidence. A model that has learned nothing useful assigns roughly uniform probability across all vocabulary items, equivalent to rolling a die with one face for each possible word. Perplexity quantifies exactly this: the effective size of the die the model is rolling at each prediction step. Lower perplexity means a smaller effective die, meaning the model has narrowed down the plausible continuations based on what it has learned.

As we discussed in earlier chapters, perplexity emerged from the n-gram era as the standard intrinsic evaluation metric. But its role has evolved significantly. Modern language models operate at vastly different scales, use subword tokenization with varying vocabulary sizes, and generate text through complex decoding strategies. These changes fundamentally alter how we calculate, interpret, and compare perplexity values. Previously we could directly compare two n-gram models trained on the same vocabulary, but today we must work with BPE merge tables, SentencePiece models, and context windows spanning thousands of tokens.

This chapter explores perplexity as an evaluation framework for contemporary language models. We will examine the mathematical foundations connecting perplexity to information theory, work through the precise calculations for both causal and masked language modeling objectives, implement evaluation pipelines that handle modern tokenization schemes, and confront the challenging limitations that arise when comparing perplexities across different model families.