BLEU Score › Worked Example

Let us walk through a concrete example to see how these components interact. Consider translating a German sentence into English, where the German original might have been "Die Katze saß auf der Matte."

Candidate: "The cat sat on the mat" Reference 1: "The cat sat on the mat" Reference 2: "The cat was sitting on the mat"

First, we calculate modified n-gram precisions.

• Unigrams ($p_1$): Every word in the candidate appears in at least one reference. "The" appears twice in the candidate and twice in Reference 1, so its clipped count is 2. All other unigrams appear once in both candidate and Reference 1. Total clipped unigrams: 6. Total unigrams: 6. $p_1 = 6/6 = 1.0$.

• Bigrams ($p_2$): The candidate contains "The cat", "cat sat", "sat on", "on the", "the mat". All five appear in Reference 1 exactly as written. $p_2 = 5/5 = 1.0$.

• Trigrams ($p_3$): "The cat sat", "cat sat on", "sat on the", "on the mat". All four appear in Reference 1. $p_3 = 4/4 = 1.0$.

• 4-grams ($p_4$): "The cat sat on", "cat sat on the", "sat on the mat". All three appear in Reference 1. $p_4 = 3/3 = 1.0$.

Since the candidate exactly matches Reference 1, all precisions are perfect. The candidate length $c = 6$ and the closest reference length is also 6, so $r = 6$. The brevity penalty is 1 (since $c = r$).

$$\begin{aligned} \text{BLEU} &= BP \cdot \exp\left(\sum_{n=1}^4 w_n \log p_n\right) \\ &= 1.0 \cdot \exp(0.25 \cdot \log 1 + 0.25 \cdot \log 1 + 0.25 \cdot \log 1 + 0.25 \cdot \log 1) && \text{(substituting values)} \\ &= 1.0 \cdot \exp(0.25 \cdot (0 + 0 + 0 + 0)) && \text{(since } \log 1 = 0\text{)} \\ &= 1.0 \cdot \exp(0) \\ &= 1.0 \cdot 1 \\ &= 1.0 \end{aligned}$$

Now consider a flawed candidate: "The cat sat on mat" (missing "the" before "mat").

• Unigrams: All five words appear in references. $p_1 = 5/5 = 1.0$.

• Bigrams: Candidate bigrams are [("The", "cat"), ("cat", "sat"), ("sat", "on"), ("on", "mat")]. Reference 1 bigrams include ("on", "the") and ("the", "mat") but not ("on", "mat"). So 3 out of 4 bigrams match. $p_2 = 3/4 = 0.75$.

• Trigrams: Candidate trigrams are [("The", "cat", "sat"), ("cat", "sat", "on"), ("sat", "on", "mat")]. Reference 1 contains the first two but not ("sat", "on", "mat"). $p_3 = 2/3 \approx 0.667$.

• 4-grams: Candidate 4-grams are [("The", "cat", "sat", "on"), ("cat", "sat", "on", "mat")]. Reference 1 contains the first but not the second. $p_4 = 1/2 = 0.5$.

Brevity penalty: candidate length 5, closest reference length 6. Since $c < r$, $BP = e^{(1-6/5)} = e^{-0.2} \approx 0.819$.

$$\begin{aligned} \text{BLEU} &= BP \cdot \exp\left(\sum_{n=1}^4 w_n \log p_n\right) \\ &= 0.819 \cdot \exp(0.25 \cdot \log 1 + 0.25 \cdot \log 0.75 + 0.25 \cdot \log 0.667 + 0.25 \cdot \log 0.5) && \text{(substituting precisions)} \\ &\approx 0.819 \cdot \exp(0.25 \cdot (0 - 0.288 - 0.405 - 0.693)) && \text{(evaluating logarithms)} \\ &\approx 0.819 \cdot \exp(-0.346) && \text{(summing and multiplying)} \\ &\approx 0.819 \cdot 0.708 && \text{(exponentiating)} \\ &\approx 0.58 \end{aligned}$$

This demonstrates how missing a single word cascades through higher-order n-grams, dramatically reducing the score. The error in the bigram "on mat" propagates into the trigram "sat on mat" and the 4-gram "cat sat on mat", creating a compounding penalty that reflects the grammatical incoherence of the missing article. This sensitivity to local context explains why BLEU effectively distinguishes between fluent and disfluent translations, even when the underlying vocabulary overlap remains high.

The example above uses a clean, well-formed sentence, but real-world BLEU computation encounters several important edge cases worth understanding.

Multiple references with different lengths. When Reference 2 ("The cat was sitting on the mat", length 7) is also considered, the closest reference to candidate length 5 is Reference 1 (length 6, difference 1), not Reference 2 (length 7, difference 2). The effective reference length $r = 6$ even though a shorter reference would reduce the brevity penalty. This "closest match" rule is important; an implementation that always uses the shortest reference would be more lenient, while one that always uses the longest would be more strict. The closest-match rule is the standard per the original paper.

The zero precision problem. If the candidate contained a 4-gram that appeared in neither reference, and no other 4-grams matched either, $p_4 = 0$ and the logarithm is undefined. The standard approach is to set the BLEU score to 0 when any precision is 0, which matches the logarithmic formulation's behavior as $p_n \to 0$. For sentence-level BLEU, this happens frequently for short sentences and is why smoothing was developed.

Case sensitivity. The original BLEU paper lowercases all tokens before comparison. This is the standard convention, as "The" and "the" should be considered the same word for evaluation purposes. Some implementations preserve case, which can lead to slightly different scores and makes comparisons across tools unreliable. Always verify that the tool you are using matches the case normalization expected by the benchmark you are comparing against.