Perplexity Evaluation › Key Parameters

Several parameters control the behavior of perplexity evaluation and directly affect both the accuracy of the resulting scores and the computational resources required. Understanding these parameters is needed for running fair evaluations and for interpreting published results.

The key parameters for perplexity evaluation are:

• stride: The step size for the sliding window when processing long sequences. A smaller stride increases overlap between windows. This provides more accurate perplexity estimates at the cost of additional computation. When the stride equals the maximum length, there is no overlap, and tokens at window boundaries receive less context than they could. When the stride is small (e.g., 128 tokens with a max_length of 512), most tokens are evaluated multiple times with rich context, but computational cost increases linearly with the overlap factor.

• max_length: The maximum sequence length processed in each forward pass, constrained by the model's context window. Sequences longer than this are processed in overlapping chunks. Setting this too large for a given model will cause out-of-memory errors or attention computation failures. Setting it smaller than necessary increases the number of forward passes required, slowing evaluation.

• batch_size: When evaluating multiple sequences simultaneously, batch processing can significantly reduce wall-clock time. However, padding shorter sequences to match the longest in the batch introduces non-prediction tokens, which must be masked out of the loss calculation using attention masks.

• special_token handling: Most tokenizers add special tokens like beginning-of-sequence (<bos>) and end-of-sequence (<eos>) tokens. Whether these tokens are included in the perplexity calculation affects the result. The common convention is to exclude the <bos> token from the loss computation (the model is not asked to predict it) but to include the <eos> token (the model should predict when the sequence ends).

Choosing appropriate values involves balancing accuracy against computational cost. For rigorous research evaluations, a small stride (256 or 512 tokens) is preferred to ensure reliable estimates. For rapid prototyping or monitoring training progress, larger strides or even simple truncation may suffice. The standard WikiText-103 benchmark is typically evaluated with stride equal to the model's context window divided by two, which provides a reasonable balance between efficiency and accuracy. When reporting perplexity, always document the stride value so others can reproduce your results.