BERTScore › Key Parameters

Understanding the key parameters for BERTScore is needed for applying the metric effectively across different domains and use cases. The choices you make regarding model selection, layer extraction, and weighting schemes can materially impact the resulting scores and their interpretation.

Which transformer layer to extract embeddings from (typically 9-10 for BERT-base) has a noticeable effect on evaluation quality. Intermediate layers capture the most transferable semantic information, balancing surface syntax with task-specific features from higher layers.

The layer selection parameter controls the depth within the transformer network from which we extract token representations. BERT-base consists of 12 transformer layers, each processing the output of the previous one. Early layers (1-4) focus primarily on local syntactic patterns and part-of-speech information. Middle layers (5-8) begin to integrate broader context and semantic relationships. Upper layers (9-12) capture high-level semantic and pragmatic features, though the final layer often becomes specialized for the pre-training objectives (masked language modeling and next sentence prediction). Research suggests that layers 9 or 10 give the best balance for semantic similarity tasks, capturing rich contextual meaning while avoiding the task-specific biases of the final layer. When using RoBERTa or other variants, optimal layers may differ slightly due to architectural variations, so experimentation with your specific domain is recommended.

The pre-trained model you choose fundamentally determines the semantic space in which similarity is measured. Different models yield different score distributions and semantic sensitivities.

The choice of pre-trained model BERT-base-uncased gives a good general-purpose baseline with reasonable computational requirements. RoBERTa models, trained with optimized hyperparameters and larger batches, often produce more reliable semantic representations. Larger variants (bert-large, roberta-large) offer richer representations but require more memory and computation. Domain-specific models such as SciBERT for scientific text, BioBERT for biomedical literature, or CodeBERT for programming languages can materially improve evaluation accuracy when working with specialized vocabulary. The casing strategy (cased vs. uncased) also matters: cased models preserve capitalization information, which can be important for distinguishing proper nouns and acronyms, while uncased models treat "Apple" and "apple" identically.

The language code for optimized library implementations (e.g., 'en' for English) automatically selects appropriate pre-trained models for the target language.

The language parameter ensures that the appropriate tokenizer and model architecture are used for the target language. While multilingual models like BERT-multilingual or XLM-RoBERTa can handle many languages, monolingual models often give better representations for specific languages. When evaluating non-English text, ensure that the model you select has been trained on sufficient data in that language to produce real embeddings. For low-resource languages, multilingual models may be the only viable option, though their semantic representations may be less precise than those for high-resource languages.

Applying IDF weighting requires decisions about the reference corpus used to compute word frequencies. Using a large, diverse corpus ensures stable IDF estimates but may not reflect the specific domain of your evaluation task. Using only the provided references can lead to unstable estimates with small sample sizes but ensures domain relevance. Some implementations offer options to use pre-computed IDF weights from standard corpora or to compute them dynamically. The choice affects whether common function words are effectively discounted and whether rare, domain-specific terms receive appropriate emphasis.

Raw BERTScore similarities tend to be high even for unrelated sentences because BERT embeddings are dense and cosine similarity rarely approaches zero between arbitrary text pairs. Baseline rescaling parameters allow you to compute scores against random sentence pairs and normalize the results, which makes it easier to distinguish between mediocre and high-quality generations. This is particularly important when comparing scores across different model choices or when establishing absolute quality thresholds.