Perplexity › Summary

Perplexity measures how well a language model predicts held-out text. It's derived from cross-entropy in information theory and can be interpreted as the effective branching factor: the average number of equally likely choices the model faces at each prediction step. Lower perplexity means the model assigns higher probability to actual text, indicating it has learned better patterns of language.

The derivation from information theory gives perplexity a solid theoretical grounding. Cross-entropy measures the number of bits needed to encode each word using the model's probabilities, and perplexity converts this to an intuitive scale by exponentiating. The geometric mean form of perplexity makes it sensitive to all individual word predictions, including rare cases where the model assigns near-zero probability to something that occurs.

Key takeaways:

• Cross-entropy measures the average bits needed to encode test data using the model's probability distribution. It equals the entropy of the true distribution plus the KL divergence between the true and model distributions.
• Perplexity equals $2^{\text{cross-entropy}}$ (or equivalently $e^{\text{cross-entropy in nats}}$), converting bits to a count of effective choices per step.
• Lower perplexity means the model assigns higher probability to actual text. Perplexity of 100 means 100 equally likely choices at each step on average.
• The branching factor interpretation: perplexity of 100 means the model has effectively reduced a vocabulary of 50,000 words to 100 equally plausible candidates at each step.
• Held-out evaluation prevents overfitting. Always evaluate on data the model has never seen.
• Same vocabulary and test set are required for fair model comparison. Different vocabulary sizes directly affect perplexity, even if model quality is the same.
• Perplexity doesn't always predict task performance, especially for tasks beyond text generation. QA, sentiment analysis, and reasoning tasks require capabilities that perplexity doesn't measure.
• OOV handling is essential when test data contains words not in training vocabulary.
• Log base choice (base 2 vs natural) affects how you express cross-entropy in bits vs nats, but the final perplexity value is identical either way.

Perplexity remains the standard intrinsic evaluation metric for language models. While it doesn't capture everything important about model quality, it provides a principled, comparable measure of a model's core capability: predicting what comes next in natural language. From n-gram models in the 1980s to transformers today, the metric has proven durable precisely because it measures something fundamental about what language models do.