BLEU Score › Summary

BLEU score evaluates text generation by measuring n-gram overlap between candidate and reference translations, combining modified precision across multiple n-gram orders with a brevity penalty to discourage short outputs. The modified precision clips n-gram counts to their maximum occurrence in any reference, preventing gaming through repetition and ensuring that candidates only receive credit for content present in the reference set.

Key implementation details distinguish corpus-level from sentence-level calculation. Corpus BLEU aggregates statistics across entire test sets before computing precision. This provides stable scores suitable for system comparison. Sentence-level BLEU computes scores independently, suffering from volatility and frequent zeros on short sentences due to the sparsity of higher-order n-grams.

The formula multiplies a brevity penalty by the geometric mean of n-gram precisions:

$$\text{BLEU} = BP \cdot \exp\left(\sum_{n=1}^N w_n \log p_n\right)$$

where:

• $\text{BLEU}$: the final BLEU score (between 0 and 1)
• $BP$: the brevity penalty factor
• $N$: the maximum n-gram order (typically 4)
• $w_n$: the weight for n-grams of order $n$ (typically uniform: $w_n = 1/N$)
• $p_n$: the modified precision for n-grams of order $n$

The brevity penalty $BP$ is defined as:

$$BP = \begin{cases} 1 & \text{if } c > r \\ e^{(1 - r/c)} & \text{if } c \leq r \end{cases}$$

where:

• $c$: the total length of the candidate translation (in words)
• $r$: the effective reference length, found by choosing the reference closest in length to the candidate

BLEU enabled the rapid progress of machine translation research through standardized, reproducible evaluation. Its limitations, including semantic blindness, word order sensitivity, and poor sentence-level correlation, have driven the development of complementary metrics like METEOR, BERTScore, and COMET. For modern research, the recommendation is to report BLEU alongside at least one semantically-aware metric and always compare against strong baselines within the same language pair and domain. The sacrebleu library provides standardized computation that ensures reproducibility across research groups.