BERTScore › Summary

BERTScore evaluates text generation by computing token-level semantic similarity using contextualized embeddings from pre-trained transformers. Rather than counting matching n-grams, it constructs a similarity matrix between reference and candidate token embeddings, then applies greedy matching to compute precision, recall, and F1 scores.

Key technical aspects include:

• Embedding extraction from intermediate transformer layers (typically layers 9-10 for BERT-base), which balance syntactic and semantic information better than either the lowest or the highest layers
• Cosine similarity computation between all pairs of reference and candidate token embeddings, creating a dense similarity matrix that captures synonym relationships and paraphrases
• Greedy alignment that independently matches each token to its most similar counterpart for precision and recall, giving linear-time computation with near-optimal matching quality
• Optional IDF weighting that discounts common function words and stresses rare, content-bearing terms, improving alignment with human judgments for domain-specific texts
• Baseline rescaling that normalizes scores against random sentence pairs, making absolute values interpretable and comparable across different model configurations

The metric captures paraphrases and synonyms that surface-form metrics miss. This gives evaluation that correlates strongly with human judgments of semantic adequacy. Variants like MoverScore extend the approach using optimal transport for global alignment, while domain-specific adaptations use specialized pre-trained models such as SciBERT, BioBERT, and CodeBERT to better capture technical vocabulary.

When comparing BERTScore to BLEU, the trade-off is clear: semantic richness and robustness to paraphrasing versus computational efficiency and lexical precision. BERTScore excels in evaluation scenarios where valid paraphrasing is expected and where single reference translations cannot cover all acceptable phrasings. BLEU retains value where exact terminology is required and where computational constraints are tight.

The limitations of BERTScore deserve equal attention: its computational cost, model dependency, inherited biases, context window constraints, and inability to assess fluency, factual accuracy, or discourse coherence. Understanding these limitations guides appropriate use. BERTScore is most useful as one component of a complete evaluation suite.

We'll build on these evaluation foundations in the next chapter, which explores Exact Match and F1 metrics for tasks requiring precise output structure, and later we'll examine how large language models themselves can serve as evaluators, pushing automatic evaluation closer to the fine-grained judgment of human experts.