BERTScore › Limitations and Impact

Despite its advantages, BERTScore introduces new challenges that practitioners must consider carefully.

The computational cost is significant: evaluating with BERTScore requires forward passes through a large transformer model, which makes it orders of magnitude slower than BLEU. This becomes prohibitive when evaluating billions of generated tokens during hyperparameter sweeps or large-scale benchmarking.

The computational overhead of BERTScore limits its applicability in resource-constrained environments. While BLEU can evaluate thousands of sentences per second on standard hardware, BERTScore processing may slow to dozens or hundreds of sentences per second, depending on model size and hardware acceleration. For research scenarios involving large generated datasets, such as evaluating the output of large language models on thousands of prompts, this slowdown can extend experiments from hours to days. GPU acceleration helps, but the memory requirements of large transformer models restrict batch sizes and increase infrastructure costs. This computational burden necessitates careful planning when incorporating BERTScore into evaluation pipelines, potentially requiring subsampling for preliminary experiments or reserved use for final evaluation stages.

BERTScore's behavior is model-dependent. Scores vary based on which pre-trained model you choose (BERT vs. RoBERTa vs. DeBERTa), which layer you extract embeddings from, and whether you use cased or uncased tokenization. This makes absolute scores difficult to interpret without calibration. A BERTScore F1 of 0.70 using BERT-base uncased layer 9 means something different than 0.70 using RoBERTa-large layer 17.

This model dependency complicates the comparison of results across studies or the establishment of universal quality thresholds. A score that indicates excellent quality in one paper using BERT-large might correspond to mediocre quality in another using a different configuration. Researchers must report detailed configuration parameters and ideally give baseline scores on standard datasets to let real comparison. The field has not yet converged on standard configurations, though RoBERTa-large with specific layer choices is emerging as a common default. Practitioners should calibrate scores against human judgments on their specific domain to establish appropriate thresholds for quality control.

The metric inherits biases from its underlying language model. If the pre-trained model associates certain demographic groups with negative sentiments or professional roles with specific genders, BERTScore may inappropriately penalize valid translations that contradict these stereotypes. Additionally, BERTScore struggles with languages or domains poorly represented in the pre-training data; evaluating rare languages or highly technical jargon may produce unreliable similarity estimates.

The bias issue is a serious ethical and methodological concern. Language models trained on web text reproduce the biases present in their training data, including gender stereotypes, racial biases, and cultural assumptions. When BERTScore evaluates translations involving demographic references or social content, it may assign lower scores to translations that use inclusive language or challenge traditional stereotypes, simply because such usage was rare in the training data. For example, a translation referring to a doctor as "she" might receive lower similarity scores if the pre-trained model associates medical professions primarily with male pronouns. Similarly, evaluations of low-resource languages or highly specialized technical domains suffer from the distributional limitations of the training data: if the model has rarely seen text in a particular language or domain, its embeddings may not adequately capture semantic relationships, leading to arbitrary or misleading similarity scores.

Context window limitations constrain BERTScore's applicability to long documents. Standard BERT models handle only 512 tokens, requiring chunking for longer texts, which may break cross-sentence dependencies. While Longformer or BigBird architectures (covered in Part XVII: Efficient Attention) can extend this range, they are not standard in BERTScore implementations.

The 512-token limit of standard BERT architectures poses practical challenges for evaluating long-form content such as document-level translation, multi-paragraph summarization, or book-length text generation. Chunking strategies, such as sliding windows or paragraph splitting, allow BERTScore to process longer texts but introduce artifacts at chunk boundaries. Coreference relationships that span across chunks, such as pronoun resolution or thematic continuity, may be lost when text is segmented. This limitation is particularly problematic for discourse-level evaluation, where the coherence and cohesion of long texts are important quality aspects. Recent developments in long-context transformers offer potential solutions, but their integration into standard BERTScore toolkits remains incomplete, and their computational requirements are even more substantial than those of standard models.

It is equally important to understand what BERTScore does not measure. The metric assesses semantic similarity at the token level but does not directly evaluate:

• Fluency: A candidate with perfect semantic content but scrambled word order may score well on BERTScore while being unreadable.
• Factual accuracy: BERTScore measures whether the candidate is semantically similar to the reference, not whether the reference itself is factually correct or whether the candidate preserves factual claims accurately.
• Discourse coherence: Token-level alignment cannot detect whether a multi-sentence summary flows logically or maintains consistent perspective across paragraphs.
• Style and tone: Two sentences can be semantically equivalent but wildly different in register, formality, or tone, and BERTScore will not distinguish between them.

Awareness of these gaps is important when designing evaluation protocols. For tasks where fluency, factual accuracy, or coherence matter, BERTScore should be complemented with other metrics or human evaluation. It is most useful as one component of a complete evaluation suite, not as a standalone replacement for human judgment.

BERTScore has fundamentally changed how researchers approach automatic evaluation. It demonstrated that semantic embeddings could replace surface statistics, paving the way for learned metrics like BLEURT and the use of large language models as evaluators (which we'll explore in upcoming chapters on model-based evaluation). In machine translation, summarization, and dialogue generation, BERTScore now regularly appears alongside traditional metrics. This gives a more fine-grained view of generation quality that aligns better with human judgments of meaning preservation.

The impact of BERTScore extends beyond its immediate use as a metric. By proving that pre-trained contextual embeddings could reliably assess semantic equivalence, BERTScore catalyzed the development of an entire family of neural evaluation metrics. It established the paradigm of using frozen pre-trained models as evaluators, an approach now extended by metrics that use prompting techniques with large language models. In the machine translation community, BERTScore is now a standard component of evaluation suites, alongside BLEU and chrF. In summarization research, it helps distinguish between extractive and abstractive approaches by crediting the latter for semantic preservation even when phrasing differs completely from source texts. The metric has also found applications in dialogue systems, data-to-text generation, and even image captioning evaluation, wherever the semantic fidelity of generated text must be assessed against reference descriptions.