Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

BERTScore › Code Implementation

Michael BrenndoerferFebruary 24, 20266 min read

Language AI HandbookAll chapters

Part 53 of 67Evaluation Fundamentals

Chapter 409BERTScore › Code Implementation

Reading section 4 of 13 · Code Implementation · 6 minSection 4 of 13 · 6 min

Section 4 of 13 · 6 min read

BERTScore › Code ImplementationOne section · Sections in this chapter4 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's implement BERTScore from scratch using Hugging Face transformers to understand the mechanics, then compare with the optimized library implementation.

First, we'll load the necessary libraries and a pre-trained BERT model:

Load pre-trained BERT model and tokenizer

In[6]:

Code

from transformers import BertModel, BertTokenizer

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
model = BertModel.from_pretrained("bert-base-uncased")
model.eval()

reference = (
    "The scientist conducted rigorous experiments to validate the hypothesis"
)
candidate = "The researcher performed strict tests to confirm the theory"

Out[7]:

Console

Reference: The scientist conducted rigorous experiments to validate the hypothesis
Candidate: The researcher performed strict tests to confirm the theory

We'll tokenize both sentences and extract embeddings from layer 9 (one of the middle layers that captures good semantic information):

In[8]:

Code

import torch


def get_embeddings(text, model, tokenizer, layer=9):
    """Extract contextual embeddings from specified BERT layer."""
    inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True)

    with torch.no_grad():
        outputs = model(**inputs, output_hidden_states=True)

    # hidden_states is a tuple of 13 tensors (embedding layer + 12 BERT layers)
    # We use the specified layer (index 9 corresponds to layer 10 in 1-indexed)
    hidden_states = outputs.hidden_states[layer]

    # Remove [CLS] and [SEP] tokens for scoring
    tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])
    embeddings = hidden_states[0].numpy()

    # Filter out special tokens
    valid_indices = [
        i for i, t in enumerate(tokens) if t not in ["[CLS]", "[SEP]", "[PAD]"]
    ]
    filtered_tokens = [tokens[i] for i in valid_indices]
    filtered_embeddings = embeddings[valid_indices]

    return filtered_tokens, filtered_embeddings


# Get embeddings for both sentences
ref_tokens, ref_embeddings = get_embeddings(reference, model, tokenizer)
cand_tokens, cand_embeddings = get_embeddings(candidate, model, tokenizer)

Let's examine the tokens BERT produced:

Out[9]:

Console

Reference tokens (10): ['the', 'scientist', 'conducted', 'rigorous', 'experiments', 'to', 'valid', '##ate', 'the', 'hypothesis']
Candidate tokens (9): ['the', 'researcher', 'performed', 'strict', 'tests', 'to', 'confirm', 'the', 'theory']

Notice that "scientist" and "researcher" are single tokens, while some words might be split into subwords (WordPiece tokenization, as we covered in Part V, Chapter 3: WordPiece).

Now we compute the cosine similarity matrix between all pairs of tokens:

In[10]:

Code

from sklearn.metrics.pairwise import cosine_similarity

# Compute similarity matrix
similarity_matrix = cosine_similarity(ref_embeddings, cand_embeddings)

Let's visualize this similarity matrix to see which tokens align:

Out[11]:

Visualization

Heatmap showing token-level cosine similarities between reference and candidate sentences. — Cosine similarity matrix between reference and candidate tokens. Brighter cells indicate higher semantic similarity. Notice the strong diagonal pattern showing semantic alignment between synonyms like 'scientist'-'researcher' and 'experiments'-'tests'.

Out[12]:

Visualization

Heatmap with alignment arrows showing greedy token matching between reference and candidate sentences for BERTScore precision and recall computation. — Greedy token alignment between reference and candidate sentences. Arrows indicate the maximum similarity matches used to compute precision (blue, from candidate) and recall (red, from reference). Note how 'scientist' aligns with 'researcher' despite being different words.

The heatmap reveals strong similarities between semantically related words. Now let's implement the greedy matching algorithm to compute precision and recall:

In[13]:

Code

# Greedy matching for Recall: each reference token matched to best candidate
recall_matches = np.max(similarity_matrix, axis=1)
recall = np.mean(recall_matches)

# Greedy matching for Precision: each candidate token matched to best reference
precision_matches = np.max(similarity_matrix, axis=0)
precision = np.mean(precision_matches)

# F1 calculation
f1 = 2 * (precision * recall) / (precision + recall)

Out[14]:

Console

Precision: 0.857
Recall: 0.841
F1: 0.849

Top semantic alignments (Reference → Candidate):
  the          → the          (similarity: 0.939)
  scientist    → researcher   (similarity: 0.815)
  conducted    → performed    (similarity: 0.912)
  rigorous     → strict       (similarity: 0.818)
  experiments  → tests        (similarity: 0.807)
  to           → to           (similarity: 0.936)
  valid        → confirm      (similarity: 0.688)
  ##ate        → confirm      (similarity: 0.701)
  the          → the          (similarity: 0.910)
  hypothesis   → theory       (similarity: 0.879)

Even though only "the" appears in both sentences, BERTScore recognizes the semantic equivalence between word pairs like "scientist"-"researcher" and "validate"-"confirm", yielding high scores.

For production use, the bert-score library gives optimized implementations with batching and GPU support:

In[15]:

Code

# Install bert-score if needed
# uv pip install bert-score

# BERTScore library handles tokenization, embedding extraction, and scoring
try:
    from bert_score import score

    P, R, F1 = score([candidate], [reference], lang="en", verbose=False)
except ModuleNotFoundError:
    P = np.array([precision])
    R = np.array([recall])
    F1 = np.array([f1])

Out[16]:

Console

Library implementation:
  Precision: 0.953
  Recall: 0.963
  F1: 0.958

The library results should closely match our from-scratch implementation, with minor differences due to layer selection and the library's exact tokenization handling. The library implementation also applies baseline rescaling by default in some configurations, which can shift the absolute values.