BLEU Metric (2002) › The Core Insight Behind BLEU

BLEU rests on a simple but powerful observation: good translations tend to use the same words and phrases as human translators would use. If a machine translation contains many of the same word sequences (n-grams) as human reference translations, it is likely to be a good translation. Conversely, if a machine translation uses completely different words and phrases, it is probably a poor translation. This insight provides a way to evaluate translation quality by measuring overlap between the machine output and human references, without requiring human judgment for each individual translation.

The metric works by examining n-grams of different lengths. An n-gram is simply a contiguous sequence of n words from the text. Unigrams are individual words, bigrams are two-word sequences, trigrams are three-word sequences, and so on. By looking at multiple n-gram lengths simultaneously, BLEU can capture both word-level accuracy (through unigrams) and phrase-level fluency (through longer n-grams). A translation might get individual words right but arrange them awkwardly, or it might have natural-sounding phrases but miss key content words. Examining multiple n-gram lengths helps BLEU distinguish between these different types of translation errors.

The mathematical formulation combines these components into a single score. BLEU calculates precision for each n-gram length, measuring what fraction of the n-grams in the machine translation also appear in the reference translations. These precision scores are then combined using a geometric mean, which has the important property that if precision is zero for any n-gram length, the entire score becomes zero. This ensures that translations must perform reasonably well at all n-gram lengths to achieve a high score.

The complete formula is:

In this formula, $BP$ represents the brevity penalty (which we'll discuss shortly), $w_n$ are weights for each n-gram length (typically set to 1/N for uniform weighting), and $p_n$ is the precision for n-grams of length $n$. Standard BLEU uses four n-gram lengths (N=4), examining unigrams, bigrams, trigrams, and 4-grams with equal weight.