Perplexity Evaluation › Comparing Perplexities Across Models
Language AI HandbookAll chapters Perplexity Evaluation › Comparing Perplexities Across ModelsOne section · Sections in this chapter7 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
One of the most common misuse patterns in language model evaluation is comparing perplexity values across different models without accounting for systematic biases. Several factors make such comparisons problematic, and understanding these confounds is needed for fair evaluation.
Vocabulary Size Effects
Models with larger vocabularies have more "opportunities" to distribute probability mass, which tends to increase perplexity. Consider two models: one with vocabulary size 10,000 and another with 50,000. The larger vocabulary model must assign probabilities across five times as many options, which makes it harder to achieve low perplexity even if it is better at modeling language structure.
This effect arises because perplexity measures the effective number of choices, and with more choices available, the model must be more confident in its top predictions to achieve the same perplexity. A model that perfectly predicts the next token 90% of the time will have lower perplexity with a small vocabulary than with a large one, simply because the denominator in the probability calculation changes.
This effect can be partially mitigated by comparing against a baseline uniform distribution. The "effective perplexity" or perplexity reduction from uniform provides a normalized metric:
where:
- : the perplexity normalized by vocabulary size
- : the raw perplexity score of the model
- : the size of the vocabulary (number of distinct tokens)
A model achieving perplexity 100 with vocabulary size 10,000 is arguably better than a model achieving perplexity 80 with vocabulary size 1,000, because the first achieves a 100× reduction from uniform while the second only achieves 12.5×.
This normalization helps account for the inherent difficulty of the prediction task given the vocabulary size. However, it assumes that all tokens in the vocabulary are equally likely under a uniform distribution, which is not true for natural language (character frequencies and word frequencies follow power laws). Therefore, this normalization provides only a rough correction, not a perfect solution.
Tokenization Granularity
As we saw in our worked example, tokenization significantly affects perplexity values. Consider two extreme cases:
-
Character-level tokenization: Sequences are long (many tokens per word), but the prediction task is easy (only ~100 characters). Perplexity tends to be very low, often in the range of 2 to 5, because the model only needs to distinguish among a small set of characters at each step.
-
Word-level tokenization: Sequences are short, but the prediction task is hard (tens of thousands of words). Perplexity tends to be high, often in the hundreds or thousands, because the model must choose among many distinct vocabulary items.
Subword tokenization sits between these extremes. But different subword algorithms (BPE vs WordPiece vs Unigram) and different merge counts (vocabulary sizes) produce different segmentations.
When comparing GPT-2 (BPE, vocab size 50,257) to T5 (SentencePiece, vocab size 32,100), we cannot directly compare their perplexity values. The GPT-2 model processes fewer tokens per word on average, giving it more "chances" to be correct, but faces a larger vocabulary at each step. These effects interact in complex ways that make direct comparison misleading.
The practical consequence is that leaderboards reporting perplexity on standard benchmarks like WikiText-103 are only meaningful when all compared models use the same tokenizer. When a new model claims "best perplexity on WikiText-103," we should immediately ask: which tokenizer? A model with a carefully optimized tokenizer can achieve dramatically lower perplexity on the same test set compared to an otherwise equivalent model with a naive tokenizer. The perplexity improvement would reflect tokenization choices, not better language modeling.
Context Length and Position Effects
Modern transformers process contexts ranging from 2,000 to over 100,000 tokens. Perplexity typically varies with position in the sequence:
-
Initial tokens: High perplexity due to limited context. The first token must be predicted with no preceding information, which makes it essentially a unigram prediction. The second token has only one previous token for context, and so on.
-
Middle tokens: Lower perplexity with full context available. Once the model has accumulated sufficient context to understand the topic and style, predictions become more accurate.
-
Long-range dependencies: Perplexity may increase if the model struggles to maintain coherence over long distances, or if the text introduces new topics or vocabulary not seen in the distant context.
When comparing models, we must ensure they are evaluated with comparable context windows. Evaluating a 4K context model on a book and comparing it to a 128K context model on the same book is unfair. The longer-context model can attend to previous chapters, giving it an advantage that the shorter-context model cannot match. The shorter model may see only the current page, while the longer model can reference characters and plot points introduced hundreds of pages earlier.
def analyze_perplexity_by_position(model, tokenizer, text, window_size=512):
"""
Analyze how perplexity varies across sequence positions.
"""
tokens = tokenizer.encode(text)
positions = []
perplexities = []
for i in range(1, len(tokens)):
# Context grows until window_size
start = max(0, i - window_size)
context = tokens[start:i]
target = tokens[i]
input_ids = torch.tensor([context])
target_id = torch.tensor([target])
with torch.no_grad():
outputs = model(input_ids)
logits = outputs.logits[0, -1, :] # Last position logits
probs = F.softmax(logits, dim=0)
target_prob = probs[target_id].item()
ppl = 1 / target_prob if target_prob > 0 else float("inf")
positions.append(i)
perplexities.append(ppl)
if i > 200: # Limit for demo
break
return positions, perplexities
The visualization typically reveals high initial perplexity that drops rapidly as context accumulates. This pattern has implications for fair evaluation. If we compare models on short sequences (e.g., sentences), we primarily measure their ability to predict without context, testing their unigram and bigram knowledge. On long sequences, we measure long-range modeling capability, including coreference resolution, topic consistency, and discourse structure. Different applications require different capabilities, so the appropriate evaluation context length depends on the intended use case.
This positional effect also explains why the sliding window stride matters for benchmark evaluation. When we use a stride smaller than the context window, most tokens are evaluated with substantial preceding context, creating perplexity estimates that reflect the model's full capabilities. When we use a stride equal to the context window (no overlap), tokens near the beginning of each window receive less context, creating artificially higher perplexity. Published results should always specify the stride used, since the same model can report substantially different perplexity scores depending on this single choice.
Domain Mismatch and Generalization
Perplexity is always measured with respect to a specific dataset. A model trained on Wikipedia and evaluated on Twitter data will show high perplexity not because it is a poor language model, but because the domains differ. The vocabulary, syntax, and style of social media differ substantially from encyclopedic prose.
When comparing models, we must distinguish between:
-
In-domain perplexity: Measured on the training distribution or similar text. This tells us how well the model has fit its training data, but may reflect overfitting if the score is too low compared to human performance or other benchmarks.
-
Out-of-domain perplexity: Measured on different genres, time periods, or languages. This tests generalization, though high perplexity may indicate either poor generalization or simply the difficulty of the new domain.
-
Transfer perplexity: Measured after fine-tuning on small amounts of target domain data. This evaluates how quickly the model can adapt to new domains, which is important for practical applications.
A reliable model should show lower perplexity degradation when moving out-of-domain compared to a brittle model, even if the absolute perplexity values are higher. The rate of perplexity increase as we move away from the training distribution is often more informative than the absolute value at any single point.
Domain mismatch also interacts with tokenization in subtle ways. A tokenizer trained on Wikipedia text will produce efficient tokenizations (short token sequences) for Wikipedia-like content but inefficient tokenizations (long token sequences) for domain-specific text that contains many out-of-vocabulary words. These out-of-vocabulary words get broken into more subword pieces, increasing the sequence length and changing the prediction difficulty in ways that further confound cross-domain comparison.
Test Set Contamination
A less-discussed but increasingly important confound is test set contamination. Modern language models are trained on enormous internet-scale corpora that almost certainly contain evaluation benchmark text. A model may achieve low perplexity on WikiText-103 not because it has learned to model language well, but because it has memorized the specific documents in the test set.
This contamination problem is difficult to detect and nearly impossible to fully remediate after the fact. Some research groups now use held-out temporal splits, training on data before a certain date and evaluating on data after, to reduce contamination. But this does not fully solve the problem. Popular evaluation benchmarks that have been public for years are likely extensively present in the training data of modern large language models, since those benchmarks were scraped from Wikipedia and other public sources that appear in standard training corpora.
The practical implication is that perplexity comparisons between models with different training data are especially suspect. A model trained on data that happened to include the test documents will show lower perplexity than a model that carefully excluded them, even if the first model generalizes worse at generalization. This is one reason the field has moved toward task-based evaluations where individual questions are harder to memorize, though task contamination poses its own challenges.