BLEU Metric (2002) › The Brevity Penalty: Preventing Gaming

One of BLEU's most clever design choices addresses a potential exploit in the precision-based metric. Without any constraint on length, a system could achieve artificially high precision scores by producing very short translations. Consider a machine that simply outputs "the" for every translation. Every unigram would match reference translations (since "the" appears in most English sentences), giving perfect unigram precision. This would be useless as a translation but could score well on a naive precision metric.

The brevity penalty solves this problem by multiplying the score by a factor that penalizes translations shorter than the references. When the candidate translation is at least as long as the reference, no penalty is applied. When the candidate is shorter, an exponential penalty reduces the score, with the penalty growing more severe as the translation becomes shorter relative to the reference.

The formula for the brevity penalty is:

Here, $c$ represents the length of the candidate translation (in words) and $r$ represents the effective reference length. When multiple reference translations are available, $r$ is chosen as the length of the reference closest to the candidate length, which provides a fairer comparison and avoids penalizing reasonable variations in translation length.

This penalty has an elegant mathematical property: it approaches zero as the translation becomes arbitrarily short, and it equals one when the translation matches the reference length. For example, a translation half the length of the reference would receive a penalty of approximately 0.37, reducing the final score substantially even if the precision were perfect.