BLEU Score › The Brevity Penalty

Modified n-gram precision alone creates a perverse incentive: shorter translations achieve higher precision scores. A candidate consisting of a single word that appears in the reference would achieve perfect precision, yet conveys almost no information. This problem, known as the "sparse translation" or "under-translation" issue, plagued early statistical systems that learned to minimize risk by producing short, safe outputs. To counteract this strategic shortening, BLEU introduces a brevity penalty (BP) that penalizes translations shorter than their references.

The brevity penalty compares the candidate length $c$ against the effective reference length $r$. The effective reference length is determined by finding, for each candidate sentence, the reference length closest to the candidate length (in absolute difference). For corpus-level BLEU, these lengths are summed across all sentences before computing the ratio. This "closest match" approach prevents the penalty from being overly harsh when multiple references exist with varying lengths, allowing the candidate to align with the most comparable reference length.

where:

• $BP$: the brevity penalty factor (ranging from 0 to 1)
• $c$: the total length of the candidate translation (in words)
• $r$: the effective reference length, defined as the length of the reference closest to $c$ in absolute difference

When the candidate is longer than the reference ($c > r$), no penalty is applied ($BP = 1$). This reflects the asymmetry in translation evaluation: while omitting content is a serious error, adding extra information (over-translation) is already penalized by the precision component, which will not match n-grams beyond what exists in the reference. When the candidate is shorter, the penalty grows exponentially as the ratio $r/c$ increases. A candidate half the length of the reference receives a penalty of $e^{-1} \approx 0.37$, while a candidate one-tenth the length receives $e^{-9} \approx 0.0001$, effectively zeroing out the score. The exponential formulation ensures that the penalty accelerates rapidly as translations become dangerously short, protecting against systems that might attempt to cheat by producing single-word outputs.

Notice the deliberate asymmetry in the penalty design. Over-translation (producing a longer output) does not receive an explicit length penalty, only an implicit precision penalty from n-grams in the candidate that do not match anything in the reference. Under-translation (producing a shorter output) receives both the precision penalty for any missed content and the explicit brevity penalty for the length difference. This asymmetry reflects the original designers' view that under-translation is the more dangerous pathology for statistical systems. Neural systems exhibit the opposite tendency, often generating long, repetitive outputs, which is why modern neural MT evaluation sometimes augments BLEU with explicit penalties for over-generation.