Perplexity › Key Parameters

Michael BrenndoerferMarch 26, 20251 min read
Language AI HandbookAll chapters
Part 2 of 67Classical Text Representations
Chapter 10Perplexity › Key Parameters
Reading section 16 of 17 · Key Parameters · 1 minSection 16 of 17 · 1 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 10 of 513

Section 16 of 17 · 1 min read

Perplexity › Key ParametersOne section · Sections in this chapter16 / 17 ↓
Sections in this chapter17
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

When computing and interpreting perplexity, these factors have the most impact:

Key parameters affecting perplexity computation and interpretation. Vocabulary size has the largest impact on raw perplexity values, which is why standard benchmarks use fixed vocabularies to ensure fair comparison.
ParameterTypical ValuesEffect
Vocabulary size10K-100K wordsLarger vocabularies increase perplexity because the model has more choices. Always compare models with the same vocabulary.
N-gram order2-5Higher orders typically reduce perplexity but require more data. Diminishing returns beyond trigrams for most corpora.
Smoothing parameter0.001-0.5Affects perplexity through probability estimates. Tune on development data, not test data.
Test set size10K+ wordsLarger test sets give more stable perplexity estimates. Small test sets may have high variance.
OOV rate< 5% idealHigh OOV rates make perplexity less meaningful. Consider subword tokenization for open-vocabulary evaluation.
Log base2 or eUsing log⁡2\log_2 gives bits; using ln⁡\ln gives nats. The intermediate cross-entropy values differ, but final perplexity is identical.