BERTScore › Cross-Lingual BERTScore

One natural extension of BERTScore is its application to cross-lingual evaluation, where the reference and candidate texts are in different languages. This is particularly relevant for machine translation: instead of comparing a French candidate against a French reference, you might compare a French candidate directly against an English source, eliminating the need for target-language references entirely.

Cross-lingual BERTScore relies on multilingual pre-trained models such as mBERT (multilingual BERT) or XLM-RoBERTa. These models are trained on text in dozens of languages simultaneously and learn representations that are, to a significant degree, language-agnostic. Words with similar meanings in different languages tend to cluster in nearby regions of the multilingual embedding space, even without explicit parallel supervision. For example, the English word "dog" and the French word "chien" are not the same token, but their contextual embeddings in mBERT are close because they appear in similar sentence contexts across the training data.

Cross-lingual BERTScore lets a fundamentally different evaluation paradigm called quality estimation (QE): assessing translation quality without access to reference translations. In QE, the only inputs are the source sentence and the machine translation output. Traditional QE systems required complex feature engineering and large amounts of annotated data. Cross-lingual BERTScore gives a surprisingly effective zero-shot baseline by measuring the semantic alignment between the source and the translation directly in the shared multilingual embedding space.

In practice, cross-lingual BERTScore works best for language pairs that are well-represented in the multilingual model's training data. For high-resource pairs like English-German or English-Chinese, the shared embedding space is well-populated and cross-lingual similarities are reliable. For low-resource pairs involving languages with little multilingual training data, the embedding alignment is weaker and scores may be less reliable. The evaluation of cross-lingual BERTScore on the WMT quality estimation tasks has shown promising results, often competitive with supervised QE systems trained on thousands of labeled examples.