BLEU Score › Corpus vs. Sentence-Level BLEU

BLEU operates at two distinct granularities with important practical differences that researchers must understand to avoid common evaluation pitfalls.

Sentence-level BLEU computes scores on individual sentence pairs. This provides fine-grained feedback useful for analyzing specific errors, but suffers from extreme volatility. A single sentence might contain no 4-grams matching the reference, yielding a zero score despite being partially correct. This "zero problem" occurs frequently because higher-order n-grams are sparse; a 10-word sentence contains only seven 4-grams, and if none happen to match the specific reference phrasing, the geometric mean collapses to zero. Sentence-level scores are also highly sensitive to reference diversity; with only one or two reference translations, legitimate variations not captured in the references receive unfairly low scores.

The zero problem is not a minor edge case. For a 10-word candidate translated into English, the probability of at least one 4-gram matching a single reference by chance is quite low, even for a good translation. This means that sentence-level BLEU-4 will report zero for many individually acceptable sentences. Researchers who aggregate sentence-level BLEU scores by averaging them often find the aggregate is dramatically lower than corpus-level BLEU on the same test set, and the discrepancy grows with shorter sentences.

Corpus-level BLEU aggregates statistics across entire test sets before computing precision. Counts of matching n-grams are summed across all sentences, as are total n-gram counts and lengths. This aggregation provides more stable, reliable scores because zeros in individual sentences are overwhelmed by matches elsewhere. Research consistently shows that corpus-level BLEU correlates significantly better with human judgment than sentence-level BLEU, with correlation coefficients often doubling or tripling when moving from sentence to corpus aggregation.

The distinction matters enormously in practice. Papineni et al.'s original paper validated BLEU at the corpus level, and the high correlation with human judgment that made the metric famous was measured at that level. When researchers later attempted to use sentence-level BLEU for tasks like minimum risk training or reward shaping in reinforcement learning, they encountered the zero-score problem and had to develop smoothing techniques to make sentence-level scoring viable.

The choice between them depends on your goal. Use corpus-level BLEU for comparing systems or tracking overall progress, as it provides the statistical stability necessary for reliable comparisons. Use sentence-level BLEU cautiously for error analysis, and consider smoothing techniques (adding small constants to zero counts) to avoid zero scores on short sentences. When using sentence-level BLEU for training objectives or reinforcement learning, specialized smoothing methods such as Lin smoothing or the addition of epsilon values become essential to maintain gradient flow.