BLEU Score › SacreBLEU and Standardization
Language AI HandbookAll chapters BLEU Score › SacreBLEU and StandardizationOne section · Sections in this chapter9 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
One of the most persistent practical problems with BLEU is that its score depends heavily on text preprocessing: how text is tokenized, whether it is lowercased, how punctuation is handled, and whether scripts that do not use spaces as word boundaries receive special treatment. Two research groups could legitimately compute BLEU on the same system output using the same references and report scores differing by several points simply because they use different tokenization tools. This comparability problem undermined the very purpose of having a standardized metric.
The sacrebleu library, introduced by Post in 2018, addresses this problem by standardizing the entire BLEU pipeline. It enforces a specific tokenization scheme (based on the Moses tokenizer), provides canonical reference sets for standard benchmarks, and appends a BLEU signature to each score that encodes all the parameters used in computation. A score reported as BLEU+case.mixed+numrefs.1+smooth.exp+tok.13a+version.1.5.1 is fully reproducible by anyone with the same library, references, and candidate translations.
SacreBLEU encodes all preprocessing choices in a "signature" appended to each score. When reporting BLEU in a paper, include the full signature so readers can reproduce your results exactly. For example: BLEU+case.mixed+numrefs.1+smooth.exp+tok.13a.
The sacrebleu library also provides access to canonical reference sets for many standard benchmarks (WMT, FLORES, etc.) directly from the library, so you do not need to manage reference files yourself. This dramatically reduces the risk of accidentally using different references than the benchmark intends.
Beyond standardization, sacrebleu introduced several smoothing methods for sentence-level evaluation. The smooth_method='exp' option (exponential smoothing) uses a geometric series to fill in zero counts for higher-order n-grams, preventing zero scores on short sentences while preserving the relative ordering of translations. Let us explore these smoothing methods:
import sacrebleu
# Short sentence that would get zero 4-gram precision without smoothing
short_candidate = "The cat sat"
short_reference = "The cat sat on the mat"
# Without smoothing (BLEU collapses to 0 due to zero 4-gram precision)
# With smoothing (reasonable score preserved)
bleu_no_smooth = sacrebleu.sentence_bleu(
short_candidate, [short_reference], smooth_method="none"
)
bleu_floor = sacrebleu.sentence_bleu(
short_candidate, [short_reference], smooth_method="floor"
)
bleu_add_k = sacrebleu.sentence_bleu(
short_candidate, [short_reference], smooth_method="add-k"
)
bleu_exp = sacrebleu.sentence_bleu(
short_candidate, [short_reference], smooth_method="exp"
)No smoothing: 36.79 Floor smoothing: 36.79 Add-k smoothing: 36.79 Exponential: 36.79
Floor smoothing adds a small constant (typically 0.01) to any zero n-gram precision before computing the logarithm. Add-k smoothing adds to both the numerator and denominator of each precision. Exponential smoothing (the sacrebleu default) applies a geometric series to gradually smooth out zero values. Each method makes different assumptions about how a zero n-gram count should be interpreted, and the choice can significantly affect sentence-level scores for short texts.