BERTScore › From Surface Form to Semantic Similarity

To understand why BERTScore is a change in evaluation, consider this example:

Reference: The scientist conducted rigorous experiments to validate the hypothesis.
Candidate: The researcher performed strict tests to confirm the theory.

A BLEU score would see almost no matching n-grams ("scientist" ≠ "researcher", "conducted" ≠ "performed", etc.), assigning a near-zero score despite the sentences being semantically identical. BERTScore, however, recognizes that "scientist" and "researcher" share similar contexts in the training data, as do "experiments" and "tests," "validate" and "confirm," and "hypothesis" and "theory."

This example illustrates a important limitation of surface-form metrics: they cannot recognize paraphrasing, synonym substitution, or morphological variation. In real-world translation and summarization tasks, competent human translators rarely produce word-for-word matches with reference texts. Instead, they convey the same meaning using different linguistic choices. Traditional metrics penalize this valid variation, creating a disconnect between automatic scores and human judgment. BERTScore bridges this gap by operating in the semantic space rather than the lexical space.

The metric works by extracting contextual embeddings from a pre-trained model (typically BERT or RoBERTa), computing a similarity matrix between all pairs of tokens from the reference and candidate, and then finding the optimal alignment between tokens. This process turns the discrete problem of comparing word sequences into a continuous problem of comparing vectors in high-dimensional space, where semantic similarity corresponds to geometric proximity.

Consider what happens when a human expert evaluates a machine translation. The expert does not check whether the candidate shares the same exact words as the reference. Instead, they read both sentences and evaluate the overall result: does the candidate preserve the meaning, tone, and information of the reference? BERTScore approximates this process computationally by using a language model's learned representations as a proxy for human semantic understanding. The model has been trained on large corpora to understand when two expressions are interchangeable, and BERTScore exploits this knowledge without any fine-tuning or task-specific supervision.

BERTScore uses the hidden states of a pre-trained transformer to obtain contextualized representations. For a sentence with tokens $x = [x_1, x_2, ..., x_m]$, we pass it through BERT to obtain hidden states:

$$H \in \mathbb{R}^{m \times d}$$

where:

• $x$: the input sentence represented as a sequence of $m$ tokens
• $m$: the number of tokens in the sentence
• $H$: the matrix of hidden states from the final BERT layer, with each row corresponding to a token
• $d$: the hidden dimension size (768 for BERT-base)

The process of extracting these embeddings involves feeding the tokenized input through the transformer's encoder stack. Each layer of the transformer refines the representations, incorporating context from surrounding tokens through the self-attention mechanism. The lower layers tend to capture surface features such as part-of-speech tags and syntactic dependencies, while the upper layers capture more abstract semantic and contextual relationships. By selecting specific layers, we can control whether we want representations that are more syntactic or more semantic in nature.

The embedding for token $x_i$ is denoted as:

$$\mathbf{h}_{x_i} \in \mathbb{R}^d$$

where:

• $x_i$: the $i$-th token in the input sequence
• $\mathbf{h}_{x_i}$: the $d$-dimensional contextual embedding vector for token $x_i$

An important subtlety arises from the subword tokenization used by BERT. As we covered in Part V: Subword Tokenization, BERT uses WordPiece tokenization, which splits uncommon words into sub-units. The word "unbelievable" might be tokenized as ["un", "##believe", "##able"], and "running" might remain a single token while "unhappiness" gets split into ["un", "##happiness"]. This means BERTScore does not operate on word-level tokens but on subword tokens, which has practical implications. When computing scores, each subword receives its own contextual embedding and participates independently in the similarity computation. Rare words that get split into subwords effectively contribute more tokens to the score computation, which may slightly overweight them relative to common single-token words.

Another subtlety involves the special tokens that BERT prepends and appends to every sequence: [CLS] at the beginning and [SEP] at the end. These tokens participate in the self-attention computation and influence the contextual representations of surrounding tokens, but they do not correspond to any semantic content in the text. BERTScore implementations filter out these special tokens before computing similarities. This keeps the score shows only the actual content tokens in each sentence.

Given a reference sentence with $m$ tokens and a candidate sentence with $n$ tokens, we construct a similarity matrix $S \in \mathbb{R}^{m \times n}$ where each entry is the cosine similarity between token embeddings:

$$\begin{aligned} S_{ij} &= \cos(\mathbf{h}_{x_i}, \mathbf{h}_{y_j}) \\ &= \frac{\mathbf{h}_{x_i} \cdot \mathbf{h}_{y_j}}{\|\mathbf{h}_{x_i}\| \|\mathbf{h}_{y_j}\|} \end{aligned}$$

where:

• $S_{ij}$: the cosine similarity between the $i$-th reference token and $j$-th candidate token
• $\mathbf{h}_{x_i}$: the contextual embedding of the $i$-th token from the reference sentence
• $\mathbf{h}_{y_j}$: the contextual embedding of the $j$-th token from the candidate sentence
• $\cdot$: the dot product between two vectors
• $\|\cdot\|$: the L2 norm (magnitude) of a vector

The cosine similarity ranges from -1 to 1, but since BERT embeddings are non-negative in practice (due to the GELU activation functions and layer normalization), the similarities typically fall between 0 and 1.

Cosine similarity is particularly well-suited for this task because it measures the angle between vectors rather than their absolute magnitude. In the context of language model embeddings, the magnitude of a vector often correlates with the confidence or specificity of the representation, while the direction encodes the semantic meaning. By normalizing out the magnitude, cosine similarity focuses purely on semantic alignment. This means that rare, specific words and common function words can be compared on equal footing based on their directional similarity in the embedding space.

Once we have the similarity matrix, we need to determine how well the candidate tokens cover the reference tokens (recall) and how much of the candidate is supported by the reference (precision). BERTScore uses a greedy matching approach:

For Recall: Each reference token $x_i$ is matched to its most similar candidate token $y_j$:

$$\text{match\_R}(x_i) = \arg\max_j S_{ij}$$

where:

• $\text{match\_R}(x_i)$: the index of the candidate token most similar to reference token $x_i$
• $\arg\max_j$: the operator that returns the index $j$ maximizing the similarity
• $S_{ij}$: the cosine similarity between reference token $x_i$ and candidate token $y_j$

For Precision: Each candidate token $y_j$ is matched to its most similar reference token $x_i$:

$$\text{match\_P}(y_j) = \arg\max_i S_{ij}$$

where:

• $\text{match\_P}(y_j)$: the index of the reference token most similar to candidate token $y_j$
• $\arg\max_i$: the operator that returns the index $i$ maximizing the similarity
• $S_{ij}$: the cosine similarity between reference token $x_i$ and candidate token $y_j$

This greedy approach differs from optimal matching algorithms like the Earth Mover's Distance used in MoverScore, but it is computationally efficient and works well in practice.

The choice of greedy matching shows a practical compromise between computational complexity and alignment quality. In an optimal matching scenario, we might want to find the global assignment of tokens that maximizes total similarity. This keeps each token in one sentence is matched to a distinct token in the other. However, such optimal matching problems, often framed as linear assignment problems, require algorithms like the Hungarian method that scale cubically with the sequence length. For long sentences with dozens of tokens, this computational overhead becomes significant, especially when evaluating thousands of sentence pairs. Greedy matching reduces this to linear complexity while still capturing the needed semantic correspondence between texts. In practice, because semantic similarity matrices tend to have clear diagonal or near-diagonal structures (where corresponding words align), the greedy approach usually finds alignments very close to the optimal solution.

The final scores aggregate these maximum similarities:

Precision measures how much of the candidate is supported by the reference:

$$P = \frac{1}{n} \sum_{j=1}^{n} \max_{i} S_{ij}$$

where:

• $P$: the precision score (average of maximum similarities for each candidate token)
• $n$: the number of tokens in the candidate sentence
• $\max_{i} S_{ij}$: the maximum similarity between candidate token $j$ and any reference token

Recall measures how much of the reference is covered by the candidate:

$$R = \frac{1}{m} \sum_{i=1}^{m} \max_{j} S_{ij}$$

where:

• $R$: the recall score (average of maximum similarities for each reference token)
• $m$: the number of tokens in the reference sentence
• $\max_{j} S_{ij}$: the maximum similarity between reference token $i$ and any candidate token

F1 balances both:

$$F_1 = 2 \cdot \frac{P \cdot R}{P + R}$$

where:

• $F_1$: the harmonic mean of precision and recall
• $P$: the precision score
• $R$: the recall score

The aggregation strategy treats each token as an independent contributor to the overall semantic content of the sentence. When we average the maximum similarities, we are in effect asking: on average, how well does each token in the candidate find a semantic counterpart in the reference (precision), and how well does each reference token find a match in the candidate (recall)? This token-level granularity allows BERTScore to give fine-grained feedback about which specific concepts are preserved or missing in the candidate text.

The use of precision, recall, and F1 mirrors the structure of information retrieval metrics and shows similar intuitions. High precision means that the candidate does not contain content unsupported by the reference, which makes it a useful check for hallucination: if the candidate invents facts not present in the reference, those tokens will have low maximum similarities and will pull down the precision score. High recall means that the candidate covers the key concepts present in the reference, which makes it a useful check for completeness: if the candidate omits important information from the reference, those reference tokens will have no good match and will reduce the recall score. F1 balances these two concerns into a single number, rewarding candidates that are both faithful and complete.

By default, BERTScore treats all tokens equally. However, we can incorporate Inverse Document Frequency (IDF) weights to give less importance to common words like "the" or "and" (recall our discussion of TF-IDF in Part II).

The motivation for IDF weighting stems from the observation that content words (nouns, verbs, adjectives) carry more semantic information than function words (determiners, prepositions, conjunctions). In the sentence "The cat sat on the mat," the words "cat," "sat," and "mat" convey the core meaning, while "the" and "on" give grammatical structure. Without weighting, a candidate sentence that correctly captures all the function words but misses the content words could reach misleadingly high similarity scores. IDF weighting addresses this by assigning higher weights to rare, informative words and lower weights to common, structural words.

With IDF weighting, the scores become:

$$P_{\text{idf}} = \frac{\sum_{j=1}^{n} \text{idf}(y_j) \cdot \max_{i} S_{ij}}{\sum_{j=1}^{n} \text{idf}(y_j)}$$

where:

• $P_{\text{idf}}$: the IDF-weighted precision score
• $\text{idf}(y_j)$: the inverse document frequency weight for candidate token $y_j$
• $\max_{i} S_{ij}$: the maximum similarity between candidate token $y_j$ and any reference token

Similarly, the IDF-weighted recall is computed as:

$$R_{\text{idf}} = \frac{\sum_{i=1}^{m} \text{idf}(x_i) \cdot \max_{j} S_{ij}}{\sum_{i=1}^{m} \text{idf}(x_i)}$$

where:

• $R_{\text{idf}}$: the IDF-weighted recall score
• $\text{idf}(x_i)$: the inverse document frequency weight for reference token $x_i$
• $\max_{j} S_{ij}$: the maximum similarity between reference token $x_i$ and any candidate token

The IDF weights are typically computed over the reference corpus, with rare words receiving higher weights. This approach ensures that correctly matching specialized terminology or rare concepts contributes more to the final score than matching common function words. In practice, IDF weighting most noticeably improves BERTScore's correlation with human judgments when evaluating texts with rich domain-specific vocabulary, such as scientific abstracts or legal documents. For general text with a mix of function and content words, the improvement is often modest but consistent.

Computing IDF requires a reference corpus, which introduces a practical consideration: the choice of corpus affects the weights. Using a large general corpus like Wikipedia produces stable, broadly applicable weights but may not reflect the term frequency patterns of your specific evaluation domain. Using only the provided references gives more domain-relevant weights but can be unstable with small sample sizes. In the original BERTScore paper, Zhang et al. recommend using the provided references as the IDF corpus and report marginal improvements in human correlation, particularly for longer texts with varied vocabulary.