BERTScore › Worked Example

Let's walk through a concrete example to see how BERTScore handles semantic equivalence that BLEU misses. We'll step through the full computation by hand so the mechanics are completely transparent before moving to code.

Consider:

• Reference: "The cat sat"
• Candidate: "The feline rested"

We'll use simplified 3-dimensional embeddings for illustration (in practice, these are 768-dimensional). The embeddings are chosen to reflect a key property of real contextual representations: synonymous words have nearly identical vectors, while unrelated words occupy very different regions of the space:

First, we compute cosine similarities between all pairs (excluding "the" for brevity):

$$\begin{aligned} \cos(\text{cat}, \text{feline}) &= \frac{\mathbf{u} \cdot \mathbf{v}}{\|\mathbf{u}\| \|\mathbf{v}\|} \\ &= \frac{0.5(0.48) + 0.2(0.22) + 0.1(0.12)}{\sqrt{0.5^2+0.2^2+0.1^2} \cdot \sqrt{0.48^2+0.22^2+0.12^2}} \\ &\approx \frac{0.296}{0.5477 \cdot 0.5415} \\ &\approx 0.99 \end{aligned}$$

where:

• $\mathbf{u}$: the embedding vector for "cat" ($[0.5, 0.2, 0.1]$)
• $\mathbf{v}$: the embedding vector for "feline" ($[0.48, 0.22, 0.12]$)
• $\cdot$: the dot product operator
• $\|\cdot\|$: the Euclidean norm (magnitude) of the vector

$$\cos(\text{cat}, \text{rested}) \approx 0.35$$ $$\cos(\text{sat}, \text{feline}) \approx 0.28$$ $$\cos(\text{sat}, \text{rested}) \approx 0.97$$

The similarity matrix (excluding "the") is:

Recall calculation:

• "cat" is matched to "feline" (0.99)
• "sat" is matched to "rested" (0.97)
• $R = (0.99 + 0.97) / 2 = 0.98$

Precision calculation:

• "feline" is matched to "cat" (0.99)
• "rested" is matched to "sat" (0.97)
• $P = (0.99 + 0.97) / 2 = 0.98$

$F_1 = 2 \cdot (0.98 \cdot 0.98) / (0.98 + 0.98) = 0.98$

Even though no content words match exactly, BERTScore recognizes the semantic equivalence and assigns a high score. BLEU would assign 0 to unigram precision for the content words (only "The" matches) and fail to capture the quality of the candidate.

Now consider a different scenario: a candidate that is too long and introduces extra information. Suppose the candidate is "The feline rested quietly in the sun." The reference tokens "cat" and "sat" still match well to "feline" and "rested," preserving high recall. But now the candidate has four extra tokens ("quietly," "in," "the," "sun") that are not present in the reference. Each of these tokens will find its best match in the reference, but those matches will be poor. This pulls down the precision score. This shows that the candidate introduced content not supported by the reference. The F1 score will therefore be lower than for the lean, exact synonym substitution.

This example illustrates the interpretive value of looking at precision and recall separately. High recall with low precision indicates a candidate that covers the reference but adds irrelevant or hallucinated content. Low recall with high precision indicates a candidate that is faithful but incomplete. F1 gives a single balanced summary for ranking purposes, but precision and recall offer diagnostic insight when debugging system outputs.

This example shows the power of distributional semantics: words that appear in similar contexts in the training data (like "cat" and "feline," or "sat" and "rested") occupy nearby regions in the embedding space. The high cosine similarities reflect this distributional proximity, letting BERTScore to recognize that these sentences describe the same event using different vocabulary.