BLEU Metric (2002) › Working Through Examples

To build intuition for how BLEU operates in practice, let's walk through the calculation for several example translations, starting with a perfect match and then examining various types of translation errors.

Consider a reference translation:

"The cat sat on the mat"

When the candidate translation is identical: "The cat sat on the mat"

We first extract and compare n-grams at different levels. The unigrams are "The", "cat", "sat", "on", "the", "mat", and all six appear in the reference. The bigrams are "The cat", "cat sat", "sat on", "on the", and "the mat", and all five appear in the reference. Similarly, the trigrams and 4-grams would all match perfectly.

The precision calculations become straightforward. For unigrams, we have 6 matches out of 6 total, giving 1.0. For bigrams, we have 5 matches out of 5 total, also giving 1.0. The same pattern holds for longer n-grams. The brevity penalty is 1.0 because the candidate length matches the reference length (both have 6 words). Combining these with equal weights in the geometric mean: $BP \times \exp(0.25 \times (\log(1.0) + \log(1.0) + \log(1.0) + \log(1.0))) = 1.0$. A perfect translation receives a perfect score.

Now consider a poor translation: "A dog ran"

This translation has completely different meaning and is much shorter. Looking at unigrams, we have "A", "dog", and "ran". The word "a" (after lowercasing) might match, but "dog" and "ran" do not appear in the reference, giving us at most 1 out of 3 matches, or precision of 0.33. For bigrams, we have "A dog" and "dog ran", neither of which appears in the reference, giving us 0 out of 2 matches, or precision of 0.0. When any n-gram precision is zero, the geometric mean becomes zero regardless of other precisions.

The brevity penalty also penalizes this translation. With a candidate length of 3 and reference length of 6, we calculate $BP = e^{(1 - 6/3)} = e^{-1} \approx 0.37$. Even if the precision weren't zero, this penalty would substantially reduce the score. The final BLEU score becomes approximately 0.0, correctly identifying this as a very poor translation.

Let's examine a more interesting case where the translation is reasonable but imperfect: "The small cat sat on a mat"

Here we have 7 words instead of 6, and we've added "small" and changed "the mat" to "a mat". The unigrams "The", "cat", "sat", "on", "a", "mat" all appear in the reference (6 out of 7 match), giving precision of 0.86. However, the bigram "small cat" doesn't appear in the reference, and neither does "a mat", so our bigram precision will be lower, perhaps 4 out of 6, or 0.67. The brevity penalty is 1.0 because the translation is not shorter than the reference. The final score would be lower than 1.0, reflecting that while the translation captures the main meaning, it includes changes not present in the reference.