BLEU Score › Modified N-gram Precision

The foundation of BLEU lies in precision calculated over n-grams of varying lengths. However, naive precision counting fails catastrophically for translation evaluation. Consider a candidate translation "the the the the the" matched against a reference "The cat sat on the mat." A simple unigram precision calculation would yield 5/5 = 100% because every word in the candidate appears in the reference, despite the candidate being complete gibberish. This pathology demonstrates why standard precision from information retrieval is inadequate for generation tasks: it assumes that the presence of correct elements constitutes success, without considering whether the output is coherent or exploits the reference through repetition.

The modification works as follows. For each n-gram type appearing in the candidate translation, we count how many times it appears in the candidate (the candidate count), then find the maximum number of times it appears in any single reference (the reference count). The clipped count is the minimum of these two values. We sum these clipped counts for all n-grams in the candidate and divide by the total number of n-grams in the candidate. This clipping mechanism is a critical safeguard against degenerate outputs. This keeps systems cannot game the metric by repeating high-frequency words like "the" or "and" to inflate their scores.

To see exactly how clipping works, consider the adversarial candidate "the the the the the" against the reference "The cat sat on the mat." The unigram "the" appears five times in the candidate (candidate count = 5), but only once in the reference (reference count = 1). After clipping, the count for "the" becomes min(5, 1) = 1. Since there are five unigrams in the candidate but only one receives credit, the clipped precision is 1/5 = 0.20. The metric correctly identifies this as a low-quality output, even though naive precision would have scored it perfectly.

Mathematically, for n-grams of order $n$, the modified precision $p_n$ aggregates clipped counts across all candidate sentences and divides by the total number of n-grams:

where:

• $p_n$: the modified precision for n-grams of order $n$ (e.g., $p_1$ for unigrams, $p_2$ for bigrams)
• $C$: an individual candidate translation in the corpus
• $\text{Candidates}$: the set of all candidate translations being evaluated
• $n\text{-gram}$: a contiguous sequence of $n$ words from the candidate
• $\text{Count}(n\text{-gram})$: the raw count of the n-gram appearing in candidate $C$
• $\text{Count}_{\text{clip}}(n\text{-gram})$: the clipped count, defined as $\min(\text{Count}(n\text{-gram}), \text{max reference count})$
• $\text{max reference count}$: the maximum number of times the n-gram appears in any single reference translation

The clipping mechanism ensures that repeated words in the candidate cannot inflate the score beyond what is supported by the references. By taking the minimum between the candidate count and the maximum reference count, we count only as many matches as exist in at least one reference. This conservative approach reflects a fundamental principle of BLEU: a candidate should only receive credit for translation content that is attested in the reference set, preventing hallucination or over-generation from being rewarded.

This modified precision is computed separately for each n-gram order (typically unigrams through 4-grams), then combined geometrically with equal weighting. The choice of 4-grams as the upper bound reflects empirical findings that longer n-grams capture essential fluency patterns while remaining robust to the inevitable variations in phrase ordering that occur across valid translations. Unigrams capture adequacy (are the right concepts present?), while 4-grams capture fluency (do the words flow together naturally?).

The decision to combine precision across four n-gram orders is not arbitrary; each order captures a qualitatively different aspect of translation quality. Understanding what each order measures helps you interpret BLEU scores and diagnose where a system falls short.

Unigram precision ($p_1$) measures vocabulary overlap: does the candidate use the right words? High unigram precision means the translation selected appropriate lexical items, even if they appear in the wrong order. A translation that scrambles every word but picks exactly the right vocabulary will score well on unigrams. Unigrams measure adequacy.

Bigram precision ($p_2$) measures the correctness of adjacent word pairs. This is where basic grammatical patterns emerge. Articles must precede their nouns ("the cat" vs. "cat the"), auxiliaries must precede their main verbs ("is running" vs. "running is"), and prepositions must follow the correct head words. A candidate can achieve high unigram precision but low bigram precision by selecting all the right words but arranging them incorrectly at the local level.

Trigram precision ($p_3$) captures short phrasal patterns: determiner-adjective-noun sequences ("the big cat"), verb-object-preposition structures ("sitting on the"), and similar three-word collocations that characterize fluent text. Languages have strong statistical preferences for particular three-word sequences; correct trigrams indicate the translation follows the syntactic and collocational patterns of the target language.

4-gram precision ($p_4$) captures longer phrasal consistency and is the most demanding requirement. Four-word matches require a phrase that is lexically correct and grammatically arranged in the specific order found in the references. Low 4-gram precision is a strong signal of non-fluency, even when the vocabulary selection is reasonable.

The geometric mean of these four values combines evidence across n-gram orders. A system that achieves perfect unigram precision but near-zero 4-gram precision is essentially a bag-of-words generator that ignores word order, and the geometric mean correctly assigns it a very low score. The exponential combination ensures that weakness at any n-gram order cannot be fully compensated by strength elsewhere.