Encoder-Decoder Framework › Evaluating Translation: BLEU Score

Evaluating sequence generation is harder than evaluating classification. You cannot just compare logits to labels because there are multiple valid translations of any source sentence. The Bilingual Evaluation Understudy (BLEU) score is the most widely used automatic metric for machine translation.

Before BLEU was proposed by Papineni et al. in 2002, evaluating machine translation required human judges to rate translation quality on a scale, an expensive and slow process. Progress in the field was measured by periodic evaluation campaigns that could take months to organize. BLEU made it possible to evaluate a system in seconds, allowing rapid iteration and system comparison. The impact on research productivity was clear: researchers could now run ablation studies and hyperparameter sweeps with automatic evaluation, dramatically accelerating progress.

The core challenge BLEU addresses is that translation quality cannot be measured by exact string match. "The cat sat on the mat" and "A cat was sitting on the mat" might both be valid translations of the same French sentence, but exact match would give the second translation a score of zero against the first as a reference. BLEU instead measures n-gram overlap, which captures the intuition that good translations share many words and phrases with human reference translations, even if the exact wording differs.

BLEU measures the n-gram overlap between the generated translation and one or more reference translations. It computes the modified precision for n-grams of orders 1 through 4, then combines them geometrically and applies a brevity penalty to discourage overly short translations.

The modified n-gram precision $p_n$ captures how many of the candidate's n-grams appear in the reference, clipped so the model cannot cheat by repeating common words. Formally:

$$p_n = \frac{\displaystyle\sum_{\text{n-gram} \in \text{candidate}} \min\!\bigl(\text{count}_{\text{cand}}(\text{n-gram}),\; \max_{\text{ref}} \text{count}_{\text{ref}}(\text{n-gram})\bigr)}{\displaystyle\sum_{\text{n-gram} \in \text{candidate}} \text{count}_{\text{cand}}(\text{n-gram})}$$

where:

• The numerator counts how many candidate n-grams appear in the references, capped by the maximum count in any single reference
• The denominator counts all n-grams in the candidate translation
• The clipping prevents the model from inflating its score by repeating high-frequency words

The brevity penalty $BP$ discourages short outputs:

$$BP = \begin{cases} 1 & \text{if } c > r \\ e^{1 - r/c} & \text{if } c \leq r \end{cases}$$

where:

• $c$ is the total length of all candidate translations
• $r$ is the reference length that best matches each candidate

The final BLEU score combines these components:

$$\text{BLEU} = BP \cdot \exp\!\left(\sum_{n=1}^{N} w_n \log p_n\right)$$

where:

• $N = 4$ uses n-grams of orders 1 through 4
• $w_n = 1/N = 0.25$ gives uniform weights across n-gram orders
• The exponential of the weighted sum equals the geometric mean of the individual precisions

BLEU scores range from 0 to 1. In practice:

• BLEU above 0.6 indicates excellent, near-human quality
• BLEU between 0.4 and 0.6 indicates good, understandable translations
• BLEU between 0.2 and 0.4 indicates decent but significant errors
• BLEU below 0.2 indicates poor quality

While BLEU is practical and widely used, it has well-known limitations worth understanding. First, BLEU is not a good metric for individual sentences: it was designed for corpus-level evaluation where it averages over thousands of examples, and single-sentence BLEU scores are noisy and unreliable. Second, BLEU can reward systems that match the surface form of a reference without capturing the meaning, and it can penalize paraphrases that are semantically equivalent but lexically different. Third, BLEU does not measure fluency directly: a translation that scrambles the order of correct n-grams might get a non-trivial unigram BLEU score even though the output is barely readable.

More recent metrics like BERTScore and COMET attempt to address these limitations by measuring semantic similarity using contextual embeddings rather than surface n-gram overlap. These metrics correlate better with human judgments but are more expensive to compute. For the purposes of this chapter, BLEU is the appropriate metric because it is interpretable and directly connected to the n-gram overlap intuition.