BERTScore › Correlation with Human Judgments

The ultimate test of any automatic evaluation metric is how well it correlates with human judgments of quality. BERTScore was extensively validated against human ratings on standard NLG benchmarks when it was introduced.

On the WMT 2017 and 2018 machine translation shared tasks, BERTScore achieved higher correlation with human adequacy judgments than BLEU, METEOR, and TER across most language pairs. The improvements were particularly pronounced for translation pairs involving languages with different word orders, such as English-Chinese and English-Japanese, where n-gram matching is inherently less reliable. For European language pairs with similar word order, the improvements over BLEU were smaller but still consistent.

On summarization benchmarks, BERTScore correlated better with human judgments of semantic adequacy and coherence than ROUGE, particularly for abstractive summaries that use diverse vocabulary relative to the source text. Extractive summaries, which copy sentences directly from the source, tended to score similarly on both metrics, since the vocabulary overlap is high regardless.

The meta-evaluation literature on evaluation metrics has also shown that no single metric perfectly captures all dimensions of human quality judgment. BERTScore correlates well with semantic adequacy (does the candidate convey the right meaning?) but less well with fluency (is the candidate grammatically correct and natural-sounding?) and coherence (does the candidate flow logically across sentences?). Combining BERTScore with metrics designed to capture these other dimensions, such as perplexity for fluency or discourse-level coherence scores, gives more complete evaluation coverage than any single metric alone.