Embedding Evaluation › Evaluation Pitfalls

Embedding evaluation is riddled with methodological pitfalls that can lead to misleading conclusions. Understanding these is as important as knowing the evaluation methods themselves. Many papers in the 2013-2020 period made strong claims about embedding quality based on methodologically flawed evaluations, and understanding why helps you avoid repeating those mistakes.

WordSim-353, one of the most cited benchmarks, mixes two different human judgments in its annotations. Some annotators rated "similarity" (do these words mean the same thing?), while others rated "relatedness" (are these words associated?). "Coffee" and "cup" are highly related but not similar; "doctor" and "physician" are both similar and related.

The dataset was split into WordSim-Similarity and WordSim-Relatedness by Agirre et al. in 2009, but most papers still report the combined score. SimLex-999 fixed this by carefully distinguishing the two in annotation instructions. A model that performs well on WordSim-353 might be capturing either similarity or relatedness (or both), which makes it hard to know what the score tells you about the model's semantic representations.

When using WordSim-353, consider reporting it alongside SimLex-999. The gap between the two scores is informative: if a model scores much higher on WordSim than SimLex, it probably captures associative relatedness better than semantic similarity. If the scores are similar, the model is capturing both dimensions.

When a word pair can't be evaluated because a word is out-of-vocabulary, most implementations simply skip it and compute correlation only on the remaining pairs. This can inflate scores because the skipped pairs tend to be harder: rare or specialized words that the model hasn't seen enough of to represent well, and that are also harder for humans to judge consistently.

The reported correlation is on an easier subset than the full benchmark. Always report the OOV rate alongside the correlation. A model that covers 95% of vocabulary and achieves 0.70 correlation is very likely better than one that covers 60% and achieves 0.75 correlation on only the pairs it can handle. The OOV rate is especially important for specialized domains: a model trained on general web text might have good coverage of general-domain benchmarks but miss many technical terms in domain-specific evaluations.

In high-dimensional spaces, a phenomenon called the hubness problem occurs: a small number of vectors become nearest neighbors of an unusually large fraction of other vectors. These "hub" words appear in top-$k$ neighbor lists far more often than expected by chance. For 300-dimensional embeddings, it's not uncommon for 1% of the vocabulary to appear in the top-10 neighbors of more than 10% of all queries.

Hubness distorts similarity metrics and can systematically bias analogy results. A hub word can appear as the false answer to many analogy questions simply because it's geometrically close to many other vectors, regardless of semantic appropriateness. Function words and very common nouns are particularly prone to becoming hubs.

Mitigation strategies include using CSLS (Cross-domain Similarity Local Scaling), which adjusts cosine similarity by subtracting the average similarity of each word to its $k$ nearest neighbors. This penalizes words that have many close neighbors (potential hubs) and rewards words with specific, targeted neighborhoods. CSLS was originally developed for cross-lingual embedding evaluation but is useful in any high-dimensional similarity task.

Perhaps the most important pitfall: the correlation between intrinsic evaluation scores and downstream task performance is surprisingly weak. Multiple papers have shown that embeddings with higher WordSim or SimLex correlations don't consistently outperform lower-scoring embeddings on NLP tasks.

The reason is that intrinsic benchmarks test specific semantic dimensions that may not matter for a given task. Named entity recognition depends more on morphological and syntactic patterns than on semantic similarity as tested by WordSim. Sentiment analysis may depend on fine-grained valence distinctions that similarity benchmarks don't test. Machine translation quality correlates more with morphological coverage and alignment properties than with synonym similarity.

This means: always evaluate on your actual downstream task if you can. Use intrinsic evaluation to understand embedding properties and guide development, but don't treat intrinsic scores as a reliable proxy for downstream task performance.

If your training corpus includes text from the domains that inspired the similarity annotations, your embeddings may score well simply because they memorized those specific relationships rather than learning general semantic representations. This is especially a concern for embeddings trained on Wikipedia, since many analogy questions in the Google Analogy Dataset derive directly from Wikipedia facts about capitals, currencies, and family relationships.

A model that can answer "Paris : France :: Berlin : ?" correctly because it saw the phrase "Berlin is the capital of Germany" in its training data hasn't necessarily learned anything general about the capital-country relation. It might fail badly on capital-country analogies involving less-famous countries whose capitals are mentioned less frequently in text.

There's no perfect solution to contamination because we rarely know exactly what text went into a large pretrained model. The best practice is to use multiple evaluation benchmarks from diverse sources and to be skeptical of unusually high scores on benchmarks that are closely related to the training domain.

The weak correlation in the figure represents a common finding in the literature: modest intrinsic scores tell you something about embedding quality, but they don't reliably predict how much the embeddings will help on any specific task. This is not an argument against intrinsic evaluation. Rather, it's an argument for using intrinsic evaluation for what it's good at: fast iteration, debugging, and understanding what properties an embedding model has, while reserving extrinsic evaluation for making final decisions about which embeddings to deploy.

A Spearman correlation of 0.72 versus 0.70 for two different models on a small benchmark looks like a meaningful difference but probably isn't. Small evaluation datasets produce noisy estimates. A difference of 0.02 in Spearman correlation on a 65-pair dataset (RG-65) is almost certainly within statistical noise.

Best practice is to report confidence intervals around correlation estimates (computable via bootstrap resampling) and to test whether differences between models are statistically significant. For analogy accuracy, the fraction correct follows a binomial distribution, so confidence intervals are straightforward to compute. Reporting "model A achieves 68.2% and model B achieves 68.5%" without confidence intervals invites incorrect conclusions about which model is better.