Perplexity Evaluation › Implementation: Computing Perplexity for Modern Models
Language AI HandbookAll chapters Perplexity Evaluation › Implementation: Computing Perplexity for Modern ModelsOne section · Sections in this chapter5 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let us implement a perplexity calculation pipeline that handles the complexities of modern transformer models, including proper handling of attention masks, aggregation strategies, and tokenization effects.
from transformers import AutoModelForCausalLM, AutoTokenizer
# Load a small model for demonstration (run once locally to cache, then comment for CI)
model_name = "gpt2"
tokenizer = AutoTokenizer.from_pretrained(
model_name
) # Run once locally, then comment out
model = AutoModelForCausalLM.from_pretrained(model_name)
model.eval() # Set to evaluation mode for inference
# Set padding token if not present
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.eos_tokenWe begin by loading GPT-2, a standard causal language model. Notice we set the padding token to the end-of-sequence token if it is not defined. This is needed for batch processing. Without a defined padding token, we cannot efficiently process multiple sequences of different lengths in parallel, which would make evaluating on large datasets impractical. Setting the padding token to the EOS token is a standard convention that allows the model to learn that the end of a sequence is distinct from padding, though in evaluation mode this primarily serves to prevent errors in the tokenizer.
def calculate_perplexity(model, tokenizer, text, stride=512, max_length=1024):
"""
Calculate perplexity using sliding window approach to handle long sequences.
For causal LMs, we cannot simply truncate because we lose context for later tokens.
Instead, we use a sliding window with a stride.
"""
import torch
encodings = tokenizer(text, return_tensors="pt")
input_ids = encodings.input_ids
seq_len = input_ids.size(1)
nlls = [] # negative log likelihoods
prev_end_loc = 0
for begin_loc in range(0, seq_len, stride):
end_loc = min(begin_loc + max_length, seq_len)
trg_len = end_loc - prev_end_loc
input_ids_chunk = input_ids[:, begin_loc:end_loc]
target_ids = input_ids_chunk.clone()
# Mask tokens we don't want to compute loss for (from previous window)
target_ids[:, :-trg_len] = -100
with torch.no_grad():
outputs = model(input_ids_chunk, labels=target_ids)
neg_log_likelihood = outputs.loss * trg_len
nlls.append(neg_log_likelihood)
prev_end_loc = end_loc
if end_loc == seq_len:
break
ppl = torch.exp(torch.stack(nlls).sum() / end_loc)
return ppl.item()This implementation uses a sliding window approach important for evaluating long sequences. Since causal models have limited context windows, we process the text in overlapping chunks. The stride parameter controls how much we shift the window each time. We accumulate the negative log-likelihoods and normalize by the total number of tokens processed.
The sliding window technique solves a basic problem in language model evaluation: we want to evaluate the model's predictions on every token, but the model can only see a limited context window. If we simply truncated the text into non-overlapping chunks, tokens at the beginning of each chunk would have less context than they should, artificially inflating their perplexity. By overlapping the windows and only counting the new tokens in each window (controlled by trg_len), we ensure that every token is predicted with the maximum available context, up to the model's context limit.
# Test with sample texts
test_texts = [
"The cat sat on the mat and looked at the mouse.",
"Quantum entanglement demonstrates non-local correlations between particles.",
"Colorless green ideas sleep furiously.",
]Perplexity Evaluation Results: ==================================================
[transformers] `loss_type=None` was set in the config but it is unrecognized. Using the default loss: `ForCausalLMLoss`.
Text: The cat sat on the mat and looked at the mouse.... Tokens: 12 | Perplexity: 34.942 Text: Quantum entanglement demonstrates non-local correl... Tokens: 13 | Perplexity: 71.719 Text: Colorless green ideas sleep furiously.... Tokens: 7 | Perplexity: 6413.259
The results demonstrate how perplexity varies with text characteristics. The simple sentence about the cat likely achieves lower perplexity because it matches patterns common in the training data. The physics sentence may have higher perplexity due to specialized vocabulary. The Chomsky sentence, while grammatically correct, is semantically anomalous and may show higher perplexity depending on how the model balances syntactic and semantic expectations.
These examples illustrate different types of linguistic difficulty. The first sentence represents prototypical language that likely appeared frequently in training. The second contains domain-specific terminology that may be less common, testing the model's handling of technical vocabulary. The third is grammatically valid but semantically unusual, testing whether the model has learned that adjectives like "colorless" and "green" are typically incompatible, or whether it relies more on syntactic patterns than semantic coherence.
import torch
import torch.nn.functional as F
def calculate_perplexity_by_token(model, tokenizer, text):
"""
Calculate per-token perplexity to see which specific tokens
the model finds surprising.
"""
inputs = tokenizer(text, return_tensors="pt")
input_ids = inputs.input_ids
with torch.no_grad():
outputs = model(input_ids, labels=input_ids)
logits = outputs.logits
# Shift logits and labels for causal LM
shift_logits = logits[..., :-1, :].contiguous()
shift_labels = input_ids[..., 1:].contiguous()
# Calculate log probabilities
log_probs = F.log_softmax(shift_logits, dim=-1)
# Gather log probs for actual tokens
token_log_probs = log_probs.gather(
2, shift_labels.unsqueeze(-1)
).squeeze(-1)
# Convert to perplexity per token
token_ppls = torch.exp(-token_log_probs)
tokens = tokenizer.convert_ids_to_tokens(shift_labels[0])
return list(zip(tokens, token_ppls[0].tolist()))Per-token perplexity breakdown: ---------------------------------------- quick | PPL: 6077.814 brown | PPL: 7363.405 fox | PPL: 523.986 jumps | PPL: 128.523 over | PPL: 15.610 the | PPL: 2.365 lazy | PPL: 1406.744 dog | PPL: 45.483 . | PPL: 11.079
Examining per-token perplexity reveals where the model struggles. Common words like "the" and "quick" typically show low perplexity (high probability), while rare or unexpected continuations spike the perplexity. The Ġ symbol represents the space character in GPT-2's byte-pair encoding, reminding us that we are evaluating at the subword level.
This granular view helps diagnose model behavior. If we see consistently high perplexity on function words like prepositions and articles, this suggests the model has not properly learned basic syntactic patterns. If content words show high perplexity while function words are low, the model may have learned grammar but lacks semantic knowledge. Spikes in perplexity at specific positions can indicate rare words, out-of-domain terminology, or contextual surprises where the model expected a different continuation.
