BLEU Score › Code Implementation
Language AI HandbookAll chapters BLEU Score › Code ImplementationOne section · Sections in this chapter8 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let us implement BLEU from scratch to understand the mechanics, then compare with standard libraries. We will calculate both sentence-level and corpus-level scores. Implementing the metric manually illuminates the importance of edge cases such as empty strings, varying reference counts, and the numerical stability of logarithmic calculations.
First, we need utilities for extracting n-grams and computing clipped counts.
from collections import Counter
from typing import List, Tuple
def get_ngrams(tokens: List[str], n: int) -> List[Tuple[str, ...]]:
"""Extract n-grams from a list of tokens."""
return [tuple(tokens[i : i + n]) for i in range(len(tokens) - n + 1)]
def clipped_count(
candidate_ngrams: Counter, reference_ngrams_list: List[Counter]
) -> int:
"""Sum of clipped counts for candidate n-grams against multiple references."""
total = 0
for ngram, count in candidate_ngrams.items():
# Max count of this n-gram in any single reference
max_ref_count = max(ref.get(ngram, 0) for ref in reference_ngrams_list)
total += min(count, max_ref_count)
return total
def modified_precision(
candidate: List[str], references: List[List[str]], n: int
) -> float:
"""Calculate modified n-gram precision."""
candidate_ngrams = Counter(get_ngrams(candidate, n))
if not candidate_ngrams:
return 0.0
reference_counters = [Counter(get_ngrams(ref, n)) for ref in references]
clipped = clipped_count(candidate_ngrams, reference_counters)
total = sum(candidate_ngrams.values())
return clipped / total if total > 0 else 0.0Now we implement the brevity penalty and the full BLEU calculation.
import math
def brevity_penalty(candidate_len: int, reference_len: int) -> float:
"""Calculate BP with effective reference length."""
if candidate_len > reference_len:
return 1.0
else:
return math.exp(1 - reference_len / candidate_len)
def closest_reference_length(
candidate: List[str], references: List[List[str]]
) -> int:
"""Find the reference length closest to candidate length."""
c_len = len(candidate)
ref_lengths = [len(ref) for ref in references]
# Find reference with length closest to candidate
closest = min(ref_lengths, key=lambda r: abs(r - c_len))
return closest
def sentence_bleu(
candidate: List[str],
references: List[List[str]],
weights: List[float] = None,
smoothing: float = 1e-10,
) -> float:
"""Calculate sentence-level BLEU."""
if weights is None:
weights = [0.25, 0.25, 0.25, 0.25] # Uniform 1-4 gram weights
max_n = len(weights)
# Calculate modified precisions for each n-gram order
precisions = []
for n in range(1, max_n + 1):
p = modified_precision(candidate, references, n)
# Smoothing: avoid log(0)
if p == 0:
p = smoothing
precisions.append(p)
# Geometric mean of precisions
log_sum = sum(w * math.log(p) for w, p in zip(weights, precisions))
geo_mean = math.exp(log_sum)
# Brevity penalty
c_len = len(candidate)
r_len = closest_reference_length(candidate, references)
bp = brevity_penalty(c_len, r_len)
return bp * geo_meanLet us test this with our earlier examples.
# Example 1: Perfect match
candidate1 = ["the", "cat", "sat", "on", "the", "mat"]
reference1 = ["the", "cat", "sat", "on", "the", "mat"]
reference2 = ["the", "cat", "was", "sitting", "on", "the", "mat"]
score1 = sentence_bleu(candidate1, [reference1, reference2])
# Example 2: Missing word
candidate2 = ["the", "cat", "sat", "on", "mat"]
score2 = sentence_bleu(candidate2, [reference1, reference2])
# Example 3: Word substitution (synonym)
candidate3 = ["the", "feline", "sat", "on", "the", "mat"]
score3 = sentence_bleu(candidate3, [reference1, reference2])
# Example 4: Adversarial repetition
candidate4 = ["the", "the", "the", "the", "the", "the"]
score4 = sentence_bleu(candidate4, [reference1, reference2])Perfect match BLEU: 1.0000 Missing 'the' BLEU: 0.5789 Synonym 'feline' BLEU: 0.5373 Adversarial repetition BLEU: 0.0000
The perfect match achieves a score near 1.0, while the missing word and synonym substitutions cause significant drops. Notice that "feline" receives a harsh penalty despite being semantically equivalent to "cat". This highlights a fundamental limitation we will discuss shortly: BLEU operates on string matching, not semantic equivalence, making it blind to paraphrase and synonymy. The adversarial repetition example shows the clipping mechanism at work: despite repeating the most common word, the score is very low because the clipped count for "the" is capped at its maximum reference occurrence.
For corpus-level BLEU, we aggregate counts across all sentences before calculating precisions. This avoids the volatility of sentence-level scoring and provides the stability necessary for reliable system comparison.
def corpus_bleu(
candidates: List[List[str]],
references_list: List[List[List[str]]],
weights: List[float] = None,
) -> float:
"""Calculate corpus-level BLEU by aggregating counts."""
if weights is None:
weights = [0.25, 0.25, 0.25, 0.25]
max_n = len(weights)
total_candidate_len = 0
total_reference_len = 0
# Aggregate counts for each n-gram order
clipped_counts = [0] * max_n
total_counts = [0] * max_n
for candidate, references in zip(candidates, references_list):
total_candidate_len += len(candidate)
total_reference_len += closest_reference_length(candidate, references)
for n in range(1, max_n + 1):
cand_ngrams = Counter(get_ngrams(candidate, n))
ref_counters = [Counter(get_ngrams(ref, n)) for ref in references]
clipped_counts[n - 1] += clipped_count(cand_ngrams, ref_counters)
total_counts[n - 1] += sum(cand_ngrams.values())
# Calculate precisions
precisions = []
for clipped, total in zip(clipped_counts, total_counts):
if total == 0:
precisions.append(0.0)
else:
precisions.append(clipped / total)
# Geometric mean
log_sum = sum(
w * math.log(p) if p > 0 else float("-inf")
for w, p in zip(weights, precisions)
)
if log_sum == float("-inf"):
geo_mean = 0.0
else:
geo_mean = math.exp(log_sum)
# Brevity penalty on corpus lengths
bp = brevity_penalty(total_candidate_len, total_reference_len)
return bp * geo_meanNow we compare sentence-level versus corpus-level scoring on a mini test set.
# Mini corpus with two sentences
candidates = [
["the", "cat", "sat", "on", "the", "mat"],
["a", "dog", "ran", "quickly"],
]
refs = [
[
["the", "cat", "sat", "on", "the", "mat"],
["the", "cat", "was", "on", "the", "mat"],
],
[["a", "dog", "ran", "fast"], ["the", "dog", "ran", "quickly"]],
]
# Sentence-level average
sent_scores = [sentence_bleu(c, r) for c, r in zip(candidates, refs)]
avg_sent = sum(sent_scores) / len(sent_scores)
# Corpus-level
corp_score = corpus_bleu(candidates, refs)Sentence-level average: 0.5016 Corpus-level BLEU: 0.9306 Individual sentence scores: ['1.0000', '0.0032']
Corpus-level BLEU typically differs from the average of sentence-level scores because it aggregates statistics globally. Notice how the second sentence, with its synonym differences ("quickly" vs "fast"), drags down the average more than it might affect corpus-level scores where other matches compensate. This mathematical property makes corpus-level evaluation the gold standard for research publications and system comparisons.
In practice, you should use established libraries like sacrebleu or nltk rather than implementing BLEU yourself, as they handle edge cases (empty strings, smoothing, multiple references) robustly and follow standardized preprocessing conventions that ensure comparability across studies.
import subprocess
# Install sacrebleu for comparison (using uv since pip is not available in venv)
subprocess.check_call(["uv", "pip", "install", "sacrebleu", "-q"])
import sacrebleu
# Ensure required imports are available for the code block
# Prepare strings for sacrebleu (expects detokenized strings)
candidate_strs = ["the cat sat on the mat", "a dog ran quickly"]
reference_strs = [
["the cat sat on the mat", "the cat was on the mat"], # refs for sentence 1
["a dog ran fast", "the dog ran quickly"], # refs for sentence 2
]
# sacrebleu expects references as list of lists (transposed)
refs_formatted = [
[ref[i] for ref in reference_strs] for i in range(len(reference_strs[0]))
]
bleu = sacrebleu.corpus_bleu(candidate_strs, refs_formatted)sacrebleu corpus BLEU: 93.06 Our implementation: 93.06
The sacrebleu score matches our manual corpus calculation when both use the same tokenization and case normalization. Note that sacrebleu reports scores on a 0-100 scale (multiply by 100) by convention, a tradition inherited from earlier machine translation competitions where integer scores were preferred for leaderboard display.