BLEU Score › Beyond BLEU: Complementary Metrics

Despite these limitations, BLEU remains relevant as a diagnostic tool. When a system shows low BLEU, it often indicates serious problems with fluency or adequacy. However, modern research increasingly supplements BLEU with semantic metrics that address its core weaknesses.

METEOR (Metric for Evaluation of Translation with Explicit ORdering) was an early attempt to address semantic blindness. It incorporates synonym matching through WordNet, stemming to handle morphological variants, and a different recall-precision trade-off that emphasizes recall more heavily than BLEU. METEOR also uses a fragmentation penalty that rewards contiguous phrase matches over scattered word matches. This provides a finer-grained assessment of word order.

TER (Translation Edit Rate) measures the number of edits required to transform the candidate into any reference translation. Edits include insertions, deletions, substitutions, and phrase shifts (reordering entire phrases). TER penalizes word order differences through shifts and is less sensitive to exact surface form matches, making it complementary to BLEU in system comparisons.

BERTScore addresses semantic blindness directly by computing similarity between candidate and reference using contextual embeddings from BERT. Rather than requiring exact string matches, BERTScore matches tokens based on their embedding similarity, allowing semantically equivalent paraphrases to receive high scores. We will explore BERTScore in detail in the next chapter, where we examine how neural representations of meaning enable more robust automatic evaluation.

COMET (Crosslingual Optimized Metric for Evaluation of Translation) takes a different approach entirely, training a neural regressor to predict human quality assessments. Given a source sentence, candidate translation, and reference, COMET produces a single quality score calibrated to human direct assessment ratings. Because it is trained to match human judgment, COMET correlates significantly better with human evaluators than BLEU, particularly at the sentence level.

The practical recommendation is to report multiple metrics. BLEU provides a computationally cheap, widely understood baseline. COMET provides the most human-aligned quality estimate. BERTScore captures semantic equivalence. Together, these three metrics give a more complete picture of system quality than any single number can convey.