Perplexity › Perplexity

You've built an n-gram language model and applied smoothing techniques. Now comes the critical question: how good is your model? Perplexity provides the answer. It's the standard metric for evaluating language models, used everywhere from academic papers to production systems. Understanding perplexity shows how to measure model quality and what it means for a model to "understand" language.

This chapter develops perplexity from first principles. We'll derive it from information theory, connect it to intuitive concepts like the "branching factor," implement it from scratch, and explore both its power and its limitations. Along the way, we'll see why perplexity has remained the dominant language model evaluation metric for over four decades, why that staying power is justified, and where the metric falls short.