BERTScore › Comparison with BLEU

As we explored in Part LIII, Chapter 3: BLEU Score, BLEU relies on precision of n-gram matches with a brevity penalty. The basic differences between these metrics illuminate the move from statistical to neural evaluation paradigms. Understanding when to prefer each metric requires appreciating what each one measures.

BLEU requires exact string matches. "Ran quickly" and "sprinted" produce zero BLEU overlap despite being synonymous. BERTScore captures these paraphrases through embedding similarity.

This distinction is the core philosophical difference between the two approaches. BLEU operates on the assumption that good translations will share specific word sequences with reference translations, an assumption that holds reasonably well for closely related languages or when multiple references are available, but breaks down for distant language pairs or creative paraphrasing. BERTScore operates on the assumption that meaning is preserved when semantic content is maintained, regardless of surface realization. This makes BERTScore more reliable to the variability inherent in human translation and summarization, where competent professionals routinely produce texts that convey the same meaning using entirely different phrasing.

BLEU operates at the n-gram level (typically up to 4-grams), while BERTScore operates at the token embedding level, effectively capturing meaning regardless of phrase boundaries or word order variations that preserve meaning.

The granularity difference affects what each metric can detect. BLEU is sensitive to local word order and collocation: "quickly ran" and "ran quickly" may receive different n-gram scores even though they are semantically equivalent. BERTScore's token-level approach focuses on semantic content rather than syntactic arrangement. This allows BERTScore to recognize that "the theory was confirmed by the researcher" and "the researcher confirmed the theory" convey the same meaning despite different syntactic structures, while BLEU might penalize the word order differences. However, this granularity also means BERTScore may miss certain types of errors that BLEU catches, such as incorrect function word usage or grammatical agreement issues that don't materially shift semantic embeddings but stand for quality problems.

BLEU is sensitive to inflections ("run" vs. "ran" vs. "running"). BERTScore handles these gracefully because contextual embeddings cluster morphological variants of the same lemma.

Morphological variation poses a significant challenge for n-gram metrics. A candidate translation using "running" where the reference uses "ran" receives no credit from BLEU, even though this is a minor grammatical variation rather than a semantic error. Stemming or lemmatization can mitigate this, but such preprocessing introduces its own errors and complexity. BERTScore naturally handles inflectional variation because pre-trained language models learn that "run," "ran," and "running" appear in similar contextual distributions, placing their embeddings in nearby regions of the vector space. This property extends to derivational morphology as well, letting BERTScore to recognize relationships between "decide," "decision," and "decisive."

BLEU requires multiple reference translations to be reliable, as a single reference cannot cover all valid paraphrases. BERTScore is more reliable with single references because it generalizes to synonymous expressions.

The cost of collecting multiple reference translations is substantial, making BLEU expensive to apply in new domains or low-resource settings. BERTScore's ability to generalize from single references materially reduces the annotation burden. A single reference sentence, combined with BERTScore's semantic generalization capabilities, can effectively evaluate a wide range of valid paraphrases that would require dozens of reference translations to cover adequately with n-gram matching. This makes BERTScore particularly useful for niche domains or specialized content where collecting multiple professional translations is prohibitively expensive.

However, BLEU retains advantages in specific scenarios. It is deterministic, model-agnostic, and computationally trivial compared to BERTScore's transformer inference. BLEU also directly measures lexical fidelity, which matters when exact terminology is required (e.g., medical or legal translations where "hypertension" cannot be paraphrased as "high blood pressure" depending on context).

In applications requiring strict terminological consistency, such as pharmaceutical documentation or patent translation, the exact word matching that BLEU enforces may be preferable to BERTScore's semantic flexibility. When translating drug labels or safety warnings, using the precise approved terminology may be legally required, and a metric that penalizes even semantically equivalent paraphrases serves a quality control function. Similarly, BLEU's computational efficiency makes it suitable for high-volume screening tasks, such as filtering candidate translations during data mining or preliminary quality assessment, where the overhead of neural inference would be prohibitive.