BERTScore › BERTScore Variants

Several extensions to the original BERTScore address specific limitations or application domains. These variants explore different alignment strategies, training paradigms, and normalization techniques to improve upon the basic approach.

While BERTScore uses greedy matching, MoverScore employs optimal transport (specifically the Earth Mover's Distance) to find the minimal cost of "moving" probability mass from candidate tokens to reference tokens. This considers the global structure of the similarity matrix rather than independent greedy matches. MoverScore often correlates better with human judgments but is computationally more expensive, scaling cubically with sequence length in the worst case compared to BERTScore's linear complexity.

The key insight behind MoverScore is that greedy matching may create suboptimal alignments when multiple tokens in one sentence could reasonably match a single token in the other. For example, in comparing "the big dog" to "the large canine," greedy matching might align both "big" and "dog" to "canine" if their similarities are locally maximal, leaving "large" unmatched even though a global view would prefer matching "big" to "large" and "dog" to "canine." Earth Mover's Distance solves this by finding the optimal flow that minimizes total transportation cost, effectively letting fractional matches and considering the entire similarity distribution. This global optimization often produces alignments that better reflect human intuition about semantic correspondence, particularly for sentences with complex syntactic structures or multiple clauses. However, the computational cost limits its applicability in high-throughput scenarios or when evaluating very long documents.

BLEURT (Bilingual Evaluation Understudy with Representations from Transformers) takes a different approach: instead of computing token-level similarities, it fine-tunes BERT on human ratings to directly predict quality scores. This requires supervised training data but can learn task-specific evaluation criteria. Unlike BERTScore, which is reference-based by design, BLEURT learns to predict human judgments end-to-end.

BLEURT is a move from unsupervised to supervised evaluation metrics. The model is fine-tuned on collections of human judgments, learning to map the embedding representations directly to quality scores. This allows BLEURT to capture fine-grained aspects of quality that might not emerge from simple similarity calculations, such as fluency, style appropriateness, or factual accuracy in specific domains. However, this supervised approach requires large labeled training data and may not generalize well to domains or tasks different from those in the training set. BLEURT also operates at the sequence level rather than the token level. This gives a single overall score rather than the detailed precision and recall breakdown offered by BERTScore. This makes BERTScore more suitable for diagnostic analysis, while BLEURT may give better absolute quality predictions when the training domain matches the evaluation domain.

Raw BERTScore similarities tend to be high (often above 0.5) even for unrelated sentences because BERT embeddings are dense and cosine similarity rarely drops to zero between arbitrary text pairs. To make scores more interpretable and comparable across different model choices, practitioners often apply baseline rescaling: computing scores against random sentences and subtracting this baseline, or using normalized scores based on empirical distributions.

Rescaling addresses the interpretability challenge posed by the absolute values of BERTScore. Because pre-trained language models produce embeddings that occupy a relatively constrained region of the high-dimensional space, even random or unrelated sentences may exhibit cosine similarities of 0.3 to 0.5. This compression of the similarity range makes it difficult to set thresholds for quality or to compare scores across different pre-trained models, which may have different baseline similarity levels. Rescaling techniques typically involve computing the mean and standard deviation of similarity scores on a corpus of unrelated sentence pairs, then z-score normalizing the actual evaluation scores against this distribution. This transformation produces scores centered around zero for random text, with positive values showing better-than-random semantic alignment and higher values showing stronger semantic equivalence.

Using domain-specific pre-trained models (SciBERT for scientific text, BioBERT for biomedical text, CodeBERT for programming languages) can improve BERTScore's sensitivity to domain terminology. For example, in evaluating code generation, CodeBERT embeddings will better capture that "append" and "push" are semantically related operations in list manipulation contexts.

Domain adaptation is important because general-purpose language models may not adequately stand for specialized vocabulary or domain-specific semantic relationships. In scientific writing, terms like "oxidation" and "reduction" have specific technical meanings distinct from their everyday usage. General BERT models might not capture the precise semantic relationships between such terms, whereas SciBERT, trained on scientific literature, encodes these relationships more accurately. Similarly, in medical contexts, BioBERT better understands that "myocardial infarction" and "heart attack" refer to the same condition, while in legal text, domain-specific models recognize that "plaintiff" and "claimant" are synonymous. For code generation evaluation, CodeBERT and similar programming-language models understand that different API calls or algorithmic approaches can reach the same computational goal, recognizing semantic equivalence between syntactically different but functionally equivalent code snippets.