BERTScore › BERTScore: Semantic Text Evaluation Using BERT Embeddings

Traditional n-gram metrics like BLEU and ROUGE dominated automatic evaluation for decades, but they share a basic limitation: they measure surface form overlap. When a candidate translation contains "feline" instead of "cat," or "automobile" instead of "car," these metrics penalize the variation despite perfect semantic equivalence. As we discussed in Part XXIV: BERT and Variants, contextualized embeddings capture meaning in a way that bag-of-words representations cannot. BERTScore applies this insight by using pre-trained contextual embeddings to evaluate text generation based on semantic similarity rather than lexical overlap.

This move from counting matching words to measuring meaning is a significant advance in how we evaluate natural language generation. Before the advent of deep contextualized representations, automatic evaluation was in effect a string matching exercise. You needed an exact word or n-gram hit to receive any credit. The key insight behind BERTScore is that pre-trained language models have already learned rich semantic relationships from large text corpora, and we can tap into that knowledge to evaluate whether two pieces of text convey the same meaning, regardless of the specific words chosen. This approach aligns automatic evaluation more closely with how humans assess quality: we care about meaning preservation, not just word overlap.

BERTScore operates on a simple yet powerful premise: if two sentences convey the same meaning, their constituent tokens should have similar contextual representations in a pre-trained language model. By computing pairwise cosine similarities between tokens in the reference and candidate texts, then aligning them greedily, BERTScore produces precision, recall, and F1 scores that correlate remarkably well with human judgments of quality. The metric was introduced by Zhang et al. in 2019 and has since become a standard component of evaluation suites across machine translation, summarization, dialogue generation, and other natural language generation tasks.

Understanding why BERTScore works requires understanding what contextual embeddings represent. Unlike static word embeddings such as Word2Vec or GloVe, which assign a single fixed vector to each word, contextual embeddings assign a different vector to each word token depending on the surrounding sentence context. The word "bank" receives one representation in "she sat by the river bank" and a completely different one in "she deposited money at the bank." This context-dependence means that contextual representations capture a word's general meaning and its sentence-specific meaning, making them ideal for semantic similarity comparisons.