Perplexity › Comparing Models with Perplexity
Language AI HandbookAll chapters Perplexity › Comparing Models with PerplexityOne section · Sections in this chapter11 / 17 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Perplexity enables fair comparison between different models, but several conditions must be satisfied for the comparison to be meaningful. Violating these conditions leads to misleading conclusions about which model is better.
Same Vocabulary Requirement
Models must use the same vocabulary for perplexity to be comparable. A model with a larger vocabulary faces a harder prediction problem because it has more choices at each step. Even if both models are equally "skilled" at understanding language patterns, the one with the larger vocabulary will show higher perplexity simply because it has more options to be uncertain among.
This is a subtle but important point. If you want to compare a model trained with a 10,000-word vocabulary to one trained with a 50,000-word vocabulary, you can't directly compare their perplexity numbers. You'd either need to evaluate both on the same restricted vocabulary, or find another normalization. This is one reason researchers often report perplexity on specific standardized benchmarks: the vocabulary is fixed, making comparisons fair across different papers and years.
# Demonstrate vocabulary effect
def create_model_with_vocab_limit(sentences, n=2, k=0.01, max_vocab=None):
"""Create model with optional vocabulary limit."""
model = NgramModel(n=n, k=k)
if max_vocab:
# Count word frequencies
word_freq = Counter()
for tokens in sentences:
word_freq.update(tokens)
# Keep only top words
top_words = set(w for w, _ in word_freq.most_common(max_vocab))
# Replace rare words with <UNK>
filtered_sentences = []
for tokens in sentences:
filtered = [w if w in top_words else "<unk>" for w in tokens]
filtered_sentences.append(filtered)
model.train(filtered_sentences)
else:
model.train(sentences)
return model
# Compare models with different vocabulary sizes
# Use sizes smaller than full vocab to show clear progression
vocab_sizes = [5, 10, 20, 35, 55, None] # None means full vocabulary
vocab_comparison = []
for max_v in vocab_sizes:
model = create_model_with_vocab_limit(
splits["train"], n=2, k=0.01, max_vocab=max_v
)
# Need to process test set with same vocabulary
if max_v:
word_freq = Counter()
for tokens in splits["train"]:
word_freq.update(tokens)
top_words = set(w for w, _ in word_freq.most_common(max_v))
test_filtered = [
[w if w in top_words else "<unk>" for w in tokens]
for tokens in splits["test"]
]
else:
test_filtered = splits["test"]
pp = model.perplexity(test_filtered)
vocab_comparison.append((max_v if max_v else len(model.vocab), pp))Effect of Vocabulary Size on Perplexity:
----------------------------------------
Vocab Size Perplexity
----------------------------------------
5 2.52
10 2.74
20 4.48
35 5.26
55 9.82
132 51.18Smaller vocabularies yield lower perplexity because the model has fewer options to choose from at each step. This illustrates why vocabulary size must be controlled when comparing models: a model with a 10-word vocabulary will always outperform one with 10,000 words on raw perplexity, even if the larger-vocabulary model is objectively better at understanding language.
Same Test Set Requirement
Perplexity scores are only comparable when computed on the same test set. Different test sets may have different inherent difficulty levels. News text is typically more predictable (lower perplexity) than tweets, poetry, or technical documentation. A model trained on news achieving perplexity 80 might be worse than a model trained on creative writing achieving perplexity 120, if we could measure them on comparable data.
This is why standard benchmark datasets like Penn Treebank (PTB), WikiText-2, and WikiText-103 exist. By providing a fixed evaluation corpus with a fixed vocabulary, these benchmarks allow researchers to compare models fairly across papers published years apart. When you see a perplexity number in an academic paper without specifying the benchmark, treat it with skepticism: it's only comparable to other numbers from the same specific dataset.
Statistical Significance
Small differences in perplexity may not be meaningful, especially when evaluated on small test sets. A difference of 2-3 perplexity points on a 10,000-word test set might just be random variation. Always consider whether improvements are statistically significant, particularly when comparing on small test sets.
The standard approach is to compute perplexity on multiple non-overlapping segments of the test set (like individual documents or sentences) and run a statistical test on the per-segment values. A paired t-test or Wilcoxon signed-rank test can tell you whether the difference between two models' perplexity scores is statistically reliable.