BERTScore › Score Interpretation and Calibration

One of the practical challenges in using BERTScore is interpreting raw scores. Unlike BLEU, which has well-established reference points (0.30 is "understandable" translation, 0.60 is near human quality), BERTScore values depend heavily on the model and configuration used. A score of 0.85 using BERT-base-uncased is not directly comparable to a score of 0.85 using RoBERTa-large.

The key insight for interpretation is that BERTScore should usually be evaluated in relative terms: which of two candidate translations scores higher for a given reference? The absolute value matters less than the ranking. When you need absolute thresholds, the recommended approach is to calibrate against human judgments on a sample of your specific domain. Compute BERTScore for a set of candidate-reference pairs that have been rated by human evaluators, then find the score threshold that best separates acceptable from unacceptable outputs in your domain.

Baseline rescaling gives a partial solution to the interpretability problem. The bert-score library supports rescaling with a pre-computed baseline: for each model and language, the library has computed the mean and standard deviation of BERTScore on a set of randomly paired sentence fragments. The rescaled score is:

where:

• $\hat{S}$: the rescaled score, centered around zero for random text
• $S$: the raw BERTScore value
• $\mu_{\text{baseline}}$: the mean BERTScore on random sentence pairs for this model
• $\sigma_{\text{baseline}}$: the standard deviation of BERTScore on random sentence pairs

After rescaling, a score near zero indicates similarity comparable to random text, positive scores indicate real semantic alignment, and scores above 1 indicate strong paraphrase-level similarity. This makes it much easier to set quality thresholds and to compare results across different studies that use the same rescaling procedure.

Even after rescaling, you should be cautious about interpreting small differences in BERTScore as real. The metric has measurement noise, particularly for short sentences where a single token alignment can materially affect the overall score. For reliable evaluation, average BERTScore over many samples and use appropriate statistical tests when comparing system outputs.