Perplexity › Per-Sentence Perplexity Analysis
Language AI HandbookAll chapters Perplexity › Per-Sentence Perplexity AnalysisOne section · Sections in this chapter9 / 17 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Aggregate perplexity hides important details. A model might perform well on average but catastrophically on certain kinds of sentences. Let's examine how perplexity varies across individual sentences to build intuition about what the metric captures.
def sentence_perplexity(model, tokens):
"""Calculate perplexity for a single sentence."""
padded = ["<s>"] * (model.n - 1) + tokens + ["</s>"]
total_log_prob = 0.0
num_tokens = 0
for i in range(model.n - 1, len(padded)):
context = tuple(padded[i - model.n + 1 : i])
word = padded[i]
total_log_prob += model.log_prob(word, context)
num_tokens += 1
avg_log_prob = total_log_prob / num_tokens
return 2 ** (-avg_log_prob)
# Analyze each test sentence with bigram model
bigram_model = models[2]
sentence_analysis = []
for tokens in test_sentences:
pp = sentence_perplexity(bigram_model, tokens)
sentence_analysis.append((" ".join(tokens), pp))Per-Sentence Perplexity (Bigram Model): ------------------------------------------------------------ PP = 2.32 "the cat sat on the mat" PP = 2.69 "the dog chased the bird" PP = 5.02 "a bird flew over the mat" PP = 19.47 "the cat and the dog played"
Sentences that closely match training patterns have lower perplexity. Novel combinations increase perplexity because the model is less certain about them. The ranking tells a story: sentences that reuse bigrams seen in training (like "the cat sat on the mat") achieve low perplexity, while sentences with entirely new word combinations (like "the cat and the dog played") have higher perplexity because those bigrams were never seen.
Per-sentence perplexity analysis is also useful for debugging. If certain sentence types consistently receive high perplexity, that points to gaps in your training data or model architecture. For example, if your language model has high perplexity on questions but low perplexity on statements, it might have been trained on a corpus that lacks interrogative sentences.
