BLEU Score › The Complete BLEU Formula

BLEU combines the brevity penalty with the geometric mean of modified n-gram precisions across different orders. The standard formulation uses 4-gram precision (weights for unigrams, bigrams, trigrams, and 4-grams), though variations exist for specific use cases. The geometric mean is chosen over the arithmetic mean because it enforces a stricter requirement: systems must perform well across all n-gram orders to achieve a high score, rather than compensating for poor performance in one order with excellence in another.

$$\text{BLEU} = BP \cdot \exp\left(\sum_{n=1}^N w_n \log p_n\right)$$

where:

• $\text{BLEU}$: the final BLEU score (between 0 and 1)
• $BP$: the brevity penalty factor
• $N$: the maximum n-gram order (typically 4)
• $w_n$: the weight for n-grams of order $n$ (typically uniform: $w_n = 1/N$)
• $p_n$: the modified precision for n-grams of order $n$
• $\exp$: the exponential function, which converts the weighted log average back to a linear scale
• $\sum_{n=1}^N$: summation over all n-gram orders from 1 to $N$

The formula combines the precisions using a geometric mean (via the logarithmic formulation), which ensures that all n-gram orders must perform well to achieve a high score. If any $p_n$ is zero, the logarithm approaches negative infinity, making the overall score zero. In practice, smoothing is applied to avoid this edge case. The weights $w_n$ allow adjusting the relative importance of different n-gram orders, though uniform weighting is standard. Some researchers adjust these weights to emphasize fluency (higher weights on 3-grams and 4-grams) or adequacy (higher weights on unigrams and bigrams) depending on their specific evaluation priorities.

The logarithmic formulation also provides a useful mathematical property: adding scores across n-gram orders in log space is equivalent to multiplying them in probability space. This means the final BLEU score is proportional to the product of all four precision values (scaled by the brevity penalty), not their sum. A translation achieving $p_1 = p_2 = p_3 = p_4 = 0.8$ would yield a geometric mean of 0.8, while a translation achieving $p_1 = 1.0$, $p_2 = 1.0$, $p_3 = 1.0$, $p_4 = 0$ would score exactly 0 because the geometric mean collapses when any factor is zero. This strictness reflects the belief that good translations must demonstrate fluency at all n-gram levels, not just word matching.

The choice of maximum n-gram order $N$ defines different BLEU variants. The standard BLEU-4 (also written BLEU) uses $N=4$ with uniform weights. Other variants occasionally appear in the literature:

BLEU-1 uses only unigram precision and is equivalent to modified unigram precision with a brevity penalty. It measures adequacy without any fluency constraint and is sometimes used to evaluate lexical coverage in tasks where word order flexibility is high, such as information extraction or certain forms of text simplification.

BLEU-2 adds bigram precision, capturing basic grammatical dependencies. Some summarization papers use BLEU-2 because summaries may legitimately reorder content, and 4-gram precision can be overly harsh when paraphrasing is encouraged.

NIST is a variant developed by the National Institute of Standards and Technology that weights n-gram matches by their information content. Common n-grams like "the cat" receive lower weight than rare n-grams like "photosynthetic apparatus". This weighting principle is more linguistically motivated than uniform weights, as correctly translating a rare technical term deserves more credit than matching a common function word. NIST also uses a different length penalty formulation.

For most machine translation benchmarks, BLEU-4 remains the default, and when researchers write "BLEU" without qualification they almost always mean BLEU-4 with corpus-level aggregation.