Embedding Evaluation: Intrinsic and Extrinsic Methods

You've trained a Word2Vec model, loaded pre-trained GloVe vectors, or fine-tuned FastText embeddings. Now comes the critical question: are these embeddings any good? And what does "good" even mean for a vector representation of a word?

This is harder than it sounds. Word embeddings are dense, high-dimensional vectors with no obvious ground truth. You can't look at the number [0.32, -0.15, 0.78, ...] and judge whether it's a good representation of "apple." You need evaluation methods that translate embedding quality into measurable quantities that correlate with real usefulness.

The challenge runs even deeper. Word embeddings are general-purpose representations intended to work across many downstream applications. A set of embeddings that excels at capturing semantic similarity between nouns might perform poorly on syntactic tasks like part-of-speech tagging, and vice versa. Any single evaluation method necessarily captures only a slice of what makes an embedding useful. This means you need a portfolio of evaluation approaches rather than a single definitive score.

Embedding evaluation has two distinct flavors. Intrinsic evaluation measures properties of the embeddings themselves: do similar words cluster together? Do vector arithmetic operations capture semantic relationships? These are fast, interpretable tests that give immediate feedback during development. Extrinsic evaluation measures the performance of a downstream NLP system that uses the embeddings: do they improve named entity recognition, sentiment analysis, or machine translation? This is the gold standard but expensive to compute.

This chapter covers both approaches in depth. We'll work through the major word similarity datasets, understand why Spearman correlation is the right metric, implement analogy evaluation and examine the 3CosAdd method in detail, visualize embeddings with t-SNE and UMAP, run a downstream evaluation, and examine the critical issue of embedding bias. Along the way, we'll highlight common pitfalls that lead researchers to draw incorrect conclusions from evaluation results.

The distinction between intrinsic and extrinsic evaluation is fundamental and shapes every evaluation decision you'll make. Understanding why each exists and what trade-offs it embodies will help you choose the right approach for any given situation.

Intrinsic evaluation tests the embeddings directly on proxy tasks designed to measure semantic quality. The logic is: if embeddings capture meaning well, words with similar meanings should have similar vectors. You can measure this by comparing cosine similarities to human judgments, or by checking whether vector arithmetic recovers known relationships. Intrinsic evaluation answers the question "do these embeddings encode meaning in a principled way?" without requiring you to build and train an entire NLP system.

Extrinsic evaluation plugs embeddings into a real NLP system and measures task performance. The logic is: embeddings are tools for downstream applications, so their quality should ultimately be judged by whether they help those applications work better. Extrinsic evaluation answers the question "do these embeddings make my system better?"

The trade-off between them is real and consequential. Intrinsic evaluation is fast, cheap, and interpretable. You can benchmark dozens of embedding models in minutes without building a downstream system. The results are immediately meaningful: a Spearman correlation of 0.75 on SimLex-999 tells you something concrete about whether the model assigns high similarity to words humans find similar.

The problem is that intrinsic evaluation can mislead you. Embeddings that score well on word similarity might still underperform on downstream tasks if the similarity axes they capture don't align with what the task needs. A model that excels at semantic similarity might be mediocre for syntactic tasks. A model with excellent analogy accuracy on capital-country pairs might still fail at morphological tasks. The proxy tasks are, at the end of the day, proxies.

Extrinsic evaluation is the ground truth. If your embeddings improve accuracy on your NER system, they're better for that task, regardless of what any similarity benchmark says. But the cost is significant: you need to build, train, and evaluate a complete downstream system, which can take hours or days. You also face the confound that downstream performance reflects both the embeddings and the downstream model architecture, training data, and hyperparameters.

Most practitioners use both: intrinsic evaluation to guide quick development iterations and extrinsic evaluation to validate before deployment. Think of intrinsic evaluation as a fast filter and extrinsic evaluation as the final judge.

The oldest and most widely used intrinsic evaluation method asks humans to rate how similar pairs of words are, then checks whether embedding cosine similarities correlate with those human judgments. The method goes back to 1965 and remains foundational today because human similarity judgments are a natural proxy for what we want from semantic representations.

The procedure starts with a dataset of word pairs, each annotated with a human similarity score. Annotators see pairs like ("cat", "kitten") or ("car", "fuel") and assign a score on a fixed scale, typically 0 to 10, where 0 means completely unrelated and 10 means identical in meaning. These judgments are then averaged across annotators to produce a consensus score for each pair.

Given a dataset of word pairs with human similarity scores, the evaluation procedure is:

1. For each word pair $(w_1, w_2)$, retrieve the embeddings $\mathbf{v}_1$ and $\mathbf{v}_2$
2. Compute cosine similarity between the two embeddings:

where $\mathbf{v}_1 \cdot \mathbf{v}_2$ is the dot product of the two vectors and $|\mathbf{v}_1|$, $|\mathbf{v}_2|$ are their Euclidean norms. Cosine similarity measures the angle between two vectors rather than their magnitude. It ranges from $-1$ (opposite directions) to $1$ (identical directions), with $0$ indicating orthogonal (unrelated) vectors. Using cosine rather than Euclidean distance is important because word vectors often vary in magnitude for reasons unrelated to semantic similarity, and dividing by the norms removes this confound.

1. Compute Spearman rank correlation $\rho$ between the list of embedding similarities and the corresponding human scores
2. Report $\rho$ as the evaluation score

Spearman correlation is preferred over Pearson because it measures rank agreement rather than linear correlation. Human similarity ratings are ordinal data: the gap between a score of 4 and a score of 5 may not represent the same psychological distance as the gap between 7 and 8. Annotators don't calibrate a linear scale; they produce rankings. Spearman correlation respects this by comparing the relative ordering of pairs rather than the absolute numeric differences.

There's a second reason to prefer Spearman: embedding cosine similarities don't necessarily align linearly with human similarity judgments even when the rank order is correct. The human scale runs from 0 to 10 with a roughly uniform prior. Cosine similarities cluster near 0 for most pairs and near 1 only for very similar words. This compression means linear correlation (Pearson) would be systematically poor even for good embeddings. Spearman sidesteps this by working with ranks.

Several benchmark datasets have become standards in the field, each designed to test different aspects of semantic representation:

• WordSim-353 (Finkelstein et al., 2001): 353 English word pairs with human similarity judgments on a 0-10 scale. One of the earliest and still widely used benchmarks. It conflates two distinct notions: semantic similarity ("car"/"automobile") and semantic relatedness ("car"/"gas"). This conflation turned out to be a significant issue because models can score well on it by capturing either relatedness or similarity.

• SimLex-999 (Hill et al., 2015): 999 word pairs specifically designed to measure semantic similarity, not relatedness. The creators explicitly told annotators to rate similarity and not association, with careful instructions distinguishing the two. "Cat" and "dog" are related but not similar; "cat" and "kitten" are both related and similar. SimLex is generally considered a harder and more meaningful benchmark than WordSim, and a model's SimLex score is a better indicator of whether it has learned true semantic equivalence.

• MEN (Bruni et al., 2014): 3000 word pairs covering both content and function words, with relatedness judgments collected via crowdsourcing. Useful for testing coverage across word types. The larger size makes correlation estimates more statistically reliable.

• RG-65 (Rubenstein and Goodenough, 1965): The granddaddy of similarity datasets, containing just 65 carefully chosen noun pairs. Small but historically important as the benchmark that established the methodology and is still used to compare modern models to results from sixty years ago.

• SimVerb-3500 (Gerz et al., 2016): 3500 verb pairs with similarity judgments. Verbs are harder to evaluate than nouns because verbal similarity is more context-dependent: "run" and "sprint" are similar in some senses but "run" and "manage" (as in "run a company") might also be considered similar in others.

Running evaluation on multiple datasets simultaneously is good practice because each captures a different aspect of semantic quality. A model might score well on MEN (relatedness) but poorly on SimLex (true similarity), which tells you something specific about what properties the model has and hasn't learned.

Let's implement word similarity evaluation from scratch:

The Spearman correlation tells you how well your embedding similarities rank-agree with human judgments. A correlation of 0.75 means the embeddings roughly agree with human intuitions about which pairs are most similar. Perfect agreement would be 1.0, and state-of-the-art models on SimLex-999 typically achieve around 0.75-0.85. Human inter-annotator agreement on SimLex is about 0.67, which provides an important reference point: no model can be expected to exceed the ceiling set by human agreement.

Reading a single Spearman correlation number is necessary but not sufficient. Several contextual factors shape what the number means. First, statistical significance matters: a correlation of 0.70 on a dataset of 65 pairs (RG-65) is much less reliable than the same correlation on 999 pairs (SimLex-999). Compute the p-value and check that it's below a meaningful threshold, typically 0.01 for word similarity benchmarks.

Second, comparisons across datasets are not apples-to-apples. WordSim-353 and SimLex-999 measure different things, and a model's relative ranking on the two datasets can be quite different. Report results on multiple benchmarks whenever possible. Third, the correlation is computed only over pairs where both words are in the vocabulary. If you skip many pairs, the reported correlation is on an unrepresentative sample.

A critical practical issue is that embeddings may not contain all words in the evaluation dataset. When a word is missing, you either skip the pair or handle it specially (with a random vector or zero vector, for example). The choice matters: always report what fraction of pairs were skipped. Embedding vocabularies trained on one corpus often miss rare or specialized words that appear in evaluation datasets.

The OOV problem interacts with score interpretation in a subtle way. Rare words tend to have noisier, less reliable embeddings when they do appear in the vocabulary, and they're the most likely to be missing when they don't. If your vocabulary covers only common words, you miss the pairs that would be hardest to evaluate correctly, which means your reported correlation is computed on an easier subset than the full benchmark. A model that covers 95% of vocabulary and achieves 0.70 correlation is almost certainly better than one that covers 60% and achieves 0.75 correlation on the pairs it can handle.

Beyond pairwise similarity, analogy tests assess whether embeddings encode structured semantic relationships rather than just proximity. The classic example is the gender analogy: the relationship between "king" and "queen" should be the same as between "man" and "woman," meaning the vector from "man" to "woman" should be approximately parallel to the vector from "king" to "queen." If embeddings have this property, vector arithmetic becomes a powerful tool for relational reasoning.

This property is remarkable because the model wasn't explicitly trained to learn relations. Word2Vec and GloVe learn from co-occurrence statistics. The fact that relational structure emerges from simple distributional learning is one of the key insights that made word embeddings such an exciting development.

The standard analogy evaluation method is 3CosAdd. Given an analogy "a : b :: c : ?", you find the word $d$ that maximizes:

where:

• $V$ is the full vocabulary
• $\mathbf{v}_a$, $\mathbf{v}_b$, $\mathbf{v}_c$, $\mathbf{v}_d$ are the embeddings of words $a$, $b$, $c$, and candidate $d$
• $\cos(\mathbf{u}, \mathbf{v}) = \frac{\mathbf{u} \cdot \mathbf{v}}{|\mathbf{u}||\mathbf{v}|}$ is the cosine similarity
• The $\arg\max$ ranges over all vocabulary words except the three query words

This scoring function is equivalent to finding $d$ whose embedding is closest to $\mathbf{v}_b - \mathbf{v}_a + \mathbf{v}_c$. The subtraction $\mathbf{v}_b - \mathbf{v}_a$ extracts the relationship vector: for "man : king", this is roughly a "royalty" direction. Adding $\mathbf{v}_c$ (e.g., "woman") to this royalty direction should land you near "queen." The three cosine terms capture this arithmetic while handling unit-normalized vectors more cleanly than raw vector arithmetic.

Importantly, the answer candidates exclude $a$, $b$, and $c$ themselves, since the trivial solution would often be one of those words. If you didn't exclude them, "king" would frequently be the top answer for the analogy "man : king :: woman : ?" simply because "king" scores highly in cosine similarity to the target vector.

The geometric intuition behind 3CosAdd is clean, but the method has known failure modes. The most important is frequency bias: very frequent words (function words, generic nouns) tend to appear as top neighbors for many queries simply because they have dense co-occurrence patterns with many other words. These words can become "hubs" that monopolize top-$k$ neighbor lists across the embedding space. When a hub word happens to be the correct analogy answer, the method looks brilliant; when it's not, the hub crowds out the right answer.

A second failure mode is the linearity assumption itself. Vector arithmetic assumes that the relationship between "man" and "woman" is the same linear transformation as the relationship between "king" and "queen." This works well for some relations (gender, country-capital, singular-plural) but breaks down for more complex semantic relationships. The analogy "Paris : France :: Berlin : Germany" works beautifully because both involve the "capital of" relation. But "good : better :: bad : worse" works because comparative form is a linear morphological operation. Relations that don't decompose linearly in the embedding space will yield poor analogy accuracy even when the embeddings are otherwise excellent.

The 3CosMul method (Levy and Goldberg, 2014) addresses some of these issues by using a multiplicative combination rather than additive:

where $\epsilon$ is a small constant to prevent division by zero. 3CosMul is less sensitive to the hub problem and often outperforms 3CosAdd on semantic analogies, though 3CosAdd remains the more commonly reported metric for historical reasons.

The most widely used analogy benchmark is the Google Analogy Dataset (Mikolov et al., 2013), containing 19,544 analogy questions in two categories:

• Semantic analogies (8,869 questions): capital-country ("Paris : France :: Berlin : Germany"), capital-world, currency, city-in-state, family relationships
• Syntactic analogies (10,675 questions): comparative adjectives ("good : better :: bad : worse"), superlatives, past tense, plural forms, nationalities

The syntactic/semantic split is analytically useful. Embeddings that score much higher on syntactic analogies than semantic ones likely capture morphological and grammatical patterns well but struggle with world knowledge. FastText, which builds subword representations, typically excels at syntactic analogies because it can represent morphological relationships even for rare words. GloVe, trained on document-level co-occurrence, often scores higher on semantic analogies because document co-occurrence patterns capture topical associations.

Let's implement the evaluation:

State-of-the-art Word2Vec models achieve around 60-70% accuracy on the Google Analogy Dataset, with generally higher accuracy on syntactic analogies than semantic ones. GloVe typically performs similarly. FastText improves on syntactic analogies due to its subword structure, which makes it better at morphological relations like singular-plural and comparative adjectives.

Analogy accuracy is an attractive metric because it seems to directly test relational reasoning. But there are important caveats. The metric is highly sensitive to vocabulary coverage: if any of the four words in an analogy is missing from the embedding vocabulary, the entire question is skipped. Models with larger vocabularies cover more questions and compute accuracy on a larger, more representative sample. Always report both the accuracy and the coverage rate.

The accuracy metric also assumes exactly one correct answer. For "Paris : France :: Berlin : ?", "Germany" is the expected answer, but "German" or "Germanic" might be reasonable alternatives. Real analogy datasets often have quirks in their expected answers that penalize valid reasoning. And because 3CosAdd searches the entire vocabulary, models with very large vocabularies face a harder search problem than models with small vocabularies, which can make accuracy numbers across models hard to compare directly.

Numbers are abstract. Visualizations make semantic structure tangible. By projecting high-dimensional embeddings to 2D, you can visually inspect whether related words cluster together and unrelated words sit apart. Visualization is most useful during development for debugging and hypothesis generation: if you notice that your animal words don't cluster as expected, you can investigate why.

t-Distributed Stochastic Neighbor Embedding (t-SNE) is the most popular dimensionality reduction method for visualizing embeddings. It works by defining a probability distribution over pairs of high-dimensional points, where nearby points have high probability of being "neighbors," and then optimizing a low-dimensional layout to match that distribution as closely as possible.

The key insight of t-SNE is what makes it different from PCA or MDS. In the high-dimensional space, t-SNE uses a Gaussian distribution to define neighborhood probabilities. In the low-dimensional projection space, it uses a Student's t-distribution with one degree of freedom. The t-distribution has heavier tails than the Gaussian. This asymmetry is intentional: it resolves the "crowding problem" that plagues other dimensionality reduction methods. In high dimensions, there's lots of room for points to be moderately close to each other. When you project to 2D, all those moderately-close points would have to collapse together, making clusters indistinguishable. The heavy-tailed t-distribution in the projection space means that moderately-close points in high dimensions are pushed apart in the projection, preserving the separation between clusters.

The key t-SNE parameters to understand:

• perplexity: Controls the effective number of neighbors each point considers. Typical values range from 5 to 50. Low perplexity emphasizes very local structure and can fragment clusters; high perplexity incorporates more global information and merges clusters that are only locally distinct. For embedding visualization, perplexity values between 15 and 30 usually work well.
• n_iter: Number of optimization iterations. Usually 1000-2000 for convergence.
• learning_rate: Step size for gradient descent. The default "auto" setting in recent sklearn versions works well in most cases.

One critical warning: t-SNE distances are not interpretable. The distances between clusters in a t-SNE plot carry no meaningful information. Two clusters that look far apart in the t-SNE might be closer in the original space than two clusters that look adjacent. Only the local topology (which points cluster with which other points within a cluster) is meaningful. This warning is frequently ignored, leading to incorrect conclusions about global embedding structure.

Uniform Manifold Approximation and Projection (UMAP) is a newer alternative to t-SNE that has become increasingly popular for its combination of speed and structure preservation. It works by constructing a fuzzy topological representation of the data and optimizing a low-dimensional layout that approximates it. The theoretical foundation involves Riemannian geometry and algebraic topology, but the practical behavior is what matters for visualization.

UMAP has several practical advantages over t-SNE:

• Speed: UMAP scales much better to large datasets. For tens of thousands of word embeddings, t-SNE can take many minutes while UMAP finishes in seconds.
• Reproducibility: UMAP produces more consistent results across runs with the same random seed.
• Global structure: The relative positions of clusters in a UMAP projection are more geometrically meaningful than in t-SNE. If two word categories are semantically distant, they'll tend to be further apart in UMAP.
• Downstream use: UMAP can also be used as a preprocessing step for other algorithms because it better preserves global geometry.

The key UMAP parameters are:

• n_neighbors: Controls the balance between local and global structure, analogous to t-SNE's perplexity. Smaller values preserve local structure; larger values capture global relationships.
• min_dist: Controls how tightly points are packed in the projection. Smaller values allow UMAP to form tight clusters; larger values spread points out.

Let's visualize our structured embeddings with both methods:

The visualization reveals semantic structure at a glance: animals, vehicles, and technology words form distinct clusters. Within each cluster, the most semantically similar words (cat/kitten, car/automobile) sit closest together. This is the kind of qualitative check that takes seconds to interpret and can catch problems that numerical scores would miss.

Both methods reveal the same underlying cluster structure. For production use with thousands of embeddings, UMAP's speed advantage becomes substantial. The choice between them often comes down to what structure you want to preserve: t-SNE for examining intra-cluster organization, UMAP for understanding inter-cluster relationships.

Projection plots are useful for quick overview, but a complementary qualitative method is direct nearest-neighbor inspection. For any word of interest, retrieve the $k$ nearest neighbors by cosine similarity and read through them. This is often more diagnostic than a scatter plot because it lets you inspect specific cases rather than aggregate structure.

Good embeddings of "doctor" should have "physician", "nurse", "surgeon" as close neighbors. If you find "patient" very close, it might indicate the model is capturing co-occurrence patterns (doctors and patients frequently co-occur) rather than semantic similarity. If you find "Doctor" (capitalized) far away, your model might be case-sensitive in unexpected ways. These kinds of qualitative checks build intuition that purely numerical evaluations miss.

Intrinsic evaluation tells you about embedding properties. Extrinsic evaluation tells you whether those properties help with NLP tasks. The standard approach is to use embeddings as features in a downstream model and measure task performance. This is where the rubber meets the road: the only question that ultimately matters is whether better embeddings lead to better systems.

The most frequently used extrinsic benchmarks are:

• Text classification: Sentiment analysis (positive/negative/neutral) and topic classification. Usually measured with accuracy or macro-averaged F1 across classes.
• Named entity recognition (NER): Identifying spans of text that refer to named entities like people, organizations, and locations. Measured with entity-level F1.
• Part-of-speech tagging: Labeling each word with its grammatical role (noun, verb, adjective, etc.). Measured with token-level accuracy.
• Semantic textual similarity (STS): Predicting how similar two sentences are, scored on a continuous scale. Measured with Pearson or Spearman correlation against human judgments, analogous to word similarity evaluation.
• Question answering: Locating or generating answers to questions given a context passage. Measured with exact match and token-overlap F1.

The choice of downstream task matters enormously for what you learn about the embeddings. Sentiment analysis is dominated by word-level semantics: knowing that "terrible" is near "awful" in the embedding space is directly useful. NER is dominated by morphological and distributional patterns: words that appear in similar syntactic positions (company names, person names) should be near each other. POS tagging is heavily syntactic: embeddings that capture grammatical function well will outperform semantically rich embeddings on this task. If you evaluate only on sentiment, you'll miss systematic weaknesses that matter for other applications.

Two protocols exist for extrinsic evaluation, and choosing between them significantly affects what the results tell you.

Frozen embeddings: Use the embeddings as fixed input features, and train only the downstream model on top of them. The embeddings themselves are never updated. This directly measures the quality of the embedding space because the downstream model cannot compensate for embedding deficiencies. If the embeddings don't encode the information the downstream task needs, no amount of clever architecture above them can recover it.

Fine-tuning: Initialize with pretrained embeddings but allow them to update during downstream training. The final performance reflects both the initial quality of the embeddings and the downstream data. Fine-tuned models almost always outperform frozen-embedding models, because fine-tuning allows the model to specialize the representations for the specific task. But this makes it harder to compare embedding models: if both achieve high downstream accuracy after fine-tuning, it may mean the downstream data was sufficient to overcome any initial embedding differences, not that the embeddings were equally good.

For pure embedding evaluation, frozen embeddings are more informative. Fine-tuning is more appropriate when you want to know the practical performance ceiling achievable with a given embedding model as initialization.

Let's implement a simple downstream evaluation using text classification:

This pipeline shows the core extrinsic evaluation workflow. In practice, you'd replace these synthetic documents with a real labeled dataset (SST-2 for sentiment, CoNLL-2003 for NER, etc.) and compare multiple embedding models side by side on the same task.

The approach in the code above, averaging word embeddings to represent a document, is called mean pooling. It's a simple and surprisingly effective baseline. Each word contributes its embedding to the average, so the document vector reflects the average semantic content of its words. Words that appear multiple times contribute proportionally more if you use a plain average, or equally if you use a unique-word average.

Mean pooling has known limitations. It ignores word order completely: "dog bites man" and "man bites dog" produce identical document embeddings. It also treats all words equally: function words like "the" and "of" contribute as much as content words, which can dilute the semantic signal. More sophisticated alternatives like weighted averaging (down-weighting common words using IDF), concatenating min/max/mean pools, or training a full sequence model on top of the embeddings address these limitations at the cost of additional complexity.

The true value of extrinsic evaluation is comparison. Running multiple embedding models through the same downstream pipeline produces directly comparable performance numbers:

The comparison makes trade-offs immediately visible: larger GloVe embeddings outperform smaller ones, FastText with subword information performs best, and all learned embeddings far exceed random baselines. Error bars from cross-validation are essential here: without them, you can't judge whether observed differences are meaningful or within random variation.

Word embeddings encode meaning and the societal biases present in their training data. This is a serious problem when embeddings are used in consequential applications like resume screening, loan approval, or content moderation.

The mechanism is straightforward: if training text contains patterns like "he is a programmer" and "she is a nurse" more often than their reverse, the model learns vector spaces where these associations are encoded geometrically. The cosine similarity between "programmer" and "he" ends up larger than between "programmer" and "she." These are not just abstract statistical artifacts; they reflect real stereotypes baked into the corpus, and they can cause real harm when deployed.

The bias problem is particularly insidious because it's invisible from intrinsic and extrinsic evaluations that don't test for it. A model can score 0.80 on SimLex-999 and 85% on sentiment classification while encoding large gender or racial biases. This means bias evaluation needs to be a deliberate, separate step in any responsible embedding assessment.

The Word Embedding Association Test (WEAT) was introduced by Caliskan et al. in 2017 and provides a formal statistical test for bias in word embeddings. It's modeled after the Implicit Association Test (IAT) from psychology, which measures how quickly people associate concepts with attributes. WEAT adapts this to vector spaces by measuring cosine similarity rather than reaction time.

WEAT compares how strongly two sets of target words (for example, male names versus female names) are associated with two sets of attribute words (for example, career words versus family words).

The per-word association score $s(w, A, B)$ measures how much a word $w$ is more associated with attribute set $A$ than attribute set $B$:

where:

• $\mathbf{w}$, $\mathbf{a}$, $\mathbf{b}$ are the embeddings of words $w$, $a$, $b$
• $\cos(\mathbf{u}, \mathbf{v})$ is cosine similarity
• $A$ and $B$ are the two attribute word sets

The WEAT effect size $d$ then measures the differential association between two target sets:

where:

• $X$ and $Y$ are the two target word sets
• The numerator is the difference in mean association scores between the two target sets
• The denominator is the pooled standard deviation of all association scores across both target sets, giving a Cohen's $d$-style standardization

An effect size near 0 indicates no differential association: both target sets are equally associated with both attribute sets. Positive values indicate that target set $X$ is more strongly associated with attribute $A$ than target set $Y$ is. The sign and magnitude mirror Cohen's $d$ conventions: values above 0.8 are considered large. Published WEAT studies on Word2Vec and GloVe consistently find effect sizes of 0.5-1.5 for gender-career bias, which is substantial by any standard.

Real WEAT studies on Word2Vec and GloVe find that European American names are more associated with pleasant words than African American names, and that male names are more associated with career words than female names. These findings sparked significant research into bias mitigation and fairness in NLP, which remains an active and important area.

WEAT provides a clean, statistically grounded measure of specific pre-defined biases. Its limitation is precisely this: you must choose the target and attribute word sets in advance, which means you can only find biases you already know to look for. A model might pass a gender-career WEAT test while still encoding age-based bias, nationality bias, or more subtle intersectional biases.

The WEAT framework also assumes that the bias is simple and linear in the embedding space. In high dimensions, biases can be distributed across many dimensions in complex, non-linear ways that WEAT doesn't capture. Models that have been explicitly debiased with linear projection methods may pass WEAT while retaining downstream discrimination, a finding documented in several follow-up studies.

Detecting bias is a prerequisite to addressing it, but debiasing is harder. The main approaches are:

• Hard debiasing (Bolukbasi et al., 2016): Identify a "gender direction" in embedding space, defined as the first principal component of the differences between paired gender-specific words (he/she, man/woman, king/queen). Project gender-neutral words to be orthogonal to this direction, removing the component of their embedding that correlates with gender. This is fast and interpretable but has the limitation of treating gender as a single linear direction in a 300-dimensional space, which is surely an oversimplification.

• Data augmentation: Train or fine-tune on data where gendered pronouns and names have been swapped, so the model sees equal exposure to both genders in all contexts. This is more principled but requires either labeled data or careful heuristics for pronoun swapping, and it doesn't address biases not mediated by explicit gender markers.

• Counterfactual data augmentation: A more targeted version where you create counterfactual training examples by swapping demographic attributes and add them to the training set, forcing the model to produce invariant representations.

• Post-hoc neutralization: Equalize distances for semantically equivalent pairs across the protected attribute. If "nurse" is closer to "she" than to "he", adjust both to be equidistant. This is clean but can produce representations that are technically equidistant while still encoding bias through other words.

Each approach has limitations. Hard debiasing can remove useful gender distinctions that are legitimately informative (actor vs actress as distinct roles) while leaving indirect biases intact through other words. There's no silver bullet, and any deployment of word embeddings in consequential settings requires ongoing monitoring and evaluation for bias.

Embedding evaluation is riddled with methodological pitfalls that can lead to misleading conclusions. Understanding these is as important as knowing the evaluation methods themselves. Many papers in the 2013-2020 period made strong claims about embedding quality based on methodologically flawed evaluations, and understanding why helps you avoid repeating those mistakes.

WordSim-353, one of the most cited benchmarks, mixes two different human judgments in its annotations. Some annotators rated "similarity" (do these words mean the same thing?), while others rated "relatedness" (are these words associated?). "Coffee" and "cup" are highly related but not similar; "doctor" and "physician" are both similar and related.

The dataset was split into WordSim-Similarity and WordSim-Relatedness by Agirre et al. in 2009, but most papers still report the combined score. SimLex-999 fixed this by carefully distinguishing the two in annotation instructions. A model that performs well on WordSim-353 might be capturing either similarity or relatedness (or both), which makes it hard to know what the score tells you about the model's semantic representations.

When using WordSim-353, consider reporting it alongside SimLex-999. The gap between the two scores is informative: if a model scores much higher on WordSim than SimLex, it probably captures associative relatedness better than semantic similarity. If the scores are similar, the model is capturing both dimensions.

When a word pair can't be evaluated because a word is out-of-vocabulary, most implementations simply skip it and compute correlation only on the remaining pairs. This can inflate scores because the skipped pairs tend to be harder: rare or specialized words that the model hasn't seen enough of to represent well, and that are also harder for humans to judge consistently.

The reported correlation is on an easier subset than the full benchmark. Always report the OOV rate alongside the correlation. A model that covers 95% of vocabulary and achieves 0.70 correlation is very likely better than one that covers 60% and achieves 0.75 correlation on only the pairs it can handle. The OOV rate is especially important for specialized domains: a model trained on general web text might have good coverage of general-domain benchmarks but miss many technical terms in domain-specific evaluations.

In high-dimensional spaces, a phenomenon called the hubness problem occurs: a small number of vectors become nearest neighbors of an unusually large fraction of other vectors. These "hub" words appear in top-$k$ neighbor lists far more often than expected by chance. For 300-dimensional embeddings, it's not uncommon for 1% of the vocabulary to appear in the top-10 neighbors of more than 10% of all queries.

Hubness distorts similarity metrics and can systematically bias analogy results. A hub word can appear as the false answer to many analogy questions simply because it's geometrically close to many other vectors, regardless of semantic appropriateness. Function words and very common nouns are particularly prone to becoming hubs.

Mitigation strategies include using CSLS (Cross-domain Similarity Local Scaling), which adjusts cosine similarity by subtracting the average similarity of each word to its $k$ nearest neighbors. This penalizes words that have many close neighbors (potential hubs) and rewards words with specific, targeted neighborhoods. CSLS was originally developed for cross-lingual embedding evaluation but is useful in any high-dimensional similarity task.

Perhaps the most important pitfall: the correlation between intrinsic evaluation scores and downstream task performance is surprisingly weak. Multiple papers have shown that embeddings with higher WordSim or SimLex correlations don't consistently outperform lower-scoring embeddings on NLP tasks.

The reason is that intrinsic benchmarks test specific semantic dimensions that may not matter for a given task. Named entity recognition depends more on morphological and syntactic patterns than on semantic similarity as tested by WordSim. Sentiment analysis may depend on fine-grained valence distinctions that similarity benchmarks don't test. Machine translation quality correlates more with morphological coverage and alignment properties than with synonym similarity.

This means: always evaluate on your actual downstream task if you can. Use intrinsic evaluation to understand embedding properties and guide development, but don't treat intrinsic scores as a reliable proxy for downstream task performance.

If your training corpus includes text from the domains that inspired the similarity annotations, your embeddings may score well simply because they memorized those specific relationships rather than learning general semantic representations. This is especially a concern for embeddings trained on Wikipedia, since many analogy questions in the Google Analogy Dataset derive directly from Wikipedia facts about capitals, currencies, and family relationships.

A model that can answer "Paris : France :: Berlin : ?" correctly because it saw the phrase "Berlin is the capital of Germany" in its training data hasn't necessarily learned anything general about the capital-country relation. It might fail badly on capital-country analogies involving less-famous countries whose capitals are mentioned less frequently in text.

There's no perfect solution to contamination because we rarely know exactly what text went into a large pretrained model. The best practice is to use multiple evaluation benchmarks from diverse sources and to be skeptical of unusually high scores on benchmarks that are closely related to the training domain.

The weak correlation in the figure represents a common finding in the literature: modest intrinsic scores tell you something about embedding quality, but they don't reliably predict how much the embeddings will help on any specific task. This is not an argument against intrinsic evaluation. Rather, it's an argument for using intrinsic evaluation for what it's good at: fast iteration, debugging, and understanding what properties an embedding model has, while reserving extrinsic evaluation for making final decisions about which embeddings to deploy.

A Spearman correlation of 0.72 versus 0.70 for two different models on a small benchmark looks like a meaningful difference but probably isn't. Small evaluation datasets produce noisy estimates. A difference of 0.02 in Spearman correlation on a 65-pair dataset (RG-65) is almost certainly within statistical noise.

Best practice is to report confidence intervals around correlation estimates (computable via bootstrap resampling) and to test whether differences between models are statistically significant. For analogy accuracy, the fraction correct follows a binomial distribution, so confidence intervals are straightforward to compute. Reporting "model A achieves 68.2% and model B achieves 68.5%" without confidence intervals invites incorrect conclusions about which model is better.

A professional embedding evaluation doesn't rely on any single metric. Good practice is to run all available evaluations and report them together, creating a complete picture of what the embeddings do well and where they struggle.

Reading this kind of report correctly requires keeping the big picture in mind. Intrinsic metrics tell you about embedding geometry. The downstream metric tells you about task utility. Coverage tells you about breadth. Bias metrics (not included here but essential) tell you about safety and fairness properties. A complete evaluation report presents all of these dimensions and interprets them jointly rather than picking whichever number looks best.

Developing good evaluation habits is as important as understanding the methods themselves. A few principles guide effective embedding evaluation practice.

Always compare to baselines. A Spearman correlation of 0.65 on SimLex-999 is meaningless without context. Is that better than random vectors? Better than a simple word frequency baseline? Better than GloVe 50d trained on Wikipedia? Every evaluation number needs a reference point. Publish baseline numbers prominently, and always compare new models to established ones on the same benchmarks.

Report every result. It's tempting to select the evaluation metric where your model looks best and report only that. This is a form of evaluation cherry-picking that produces misleading science. Good practice is to commit to an evaluation protocol before seeing results and then report all metrics regardless of which model "wins."

Match evaluation to deployment context. If you're building a biomedical NLP system, evaluate on biomedical benchmarks (BioCreative, MedQA, clinical NER datasets), not just on general-domain benchmarks. General-domain evaluation scores are nearly useless for predicting performance on specialized text. The domain mismatch between training data, evaluation benchmarks, and deployment data is a persistent source of surprise performance drops in production.

Understand the human ceiling. Evaluation benchmarks have a natural upper bound set by human inter-annotator agreement. On SimLex-999, human agreement is about 0.67. A model scoring 0.75 is not obviously worse than a model scoring 0.78, since both are above the human agreement ceiling and the difference may be statistical noise. Understanding ceilings prevents you from drawing strong conclusions from small differences between good models.

Visualize before and after debiasing. WEAT provides numbers, but visualizing the embedding space before and after debiasing interventions often reveals whether the intervention worked or just moved the bias around. Project gendered word pairs before and after debiasing and check whether the geometric relationships changed as intended.

Embedding evaluation is a field still working out its own foundations. The measures we use are imperfect proxies, and interpreting them requires care.

The biggest limitation is that no single evaluation captures all aspects of embedding quality. Word similarity tests semantic nearness but not syntactic properties. Analogy accuracy measures structured semantic relationships but is sensitive to the specific analogy categories included. Downstream evaluation is most reliable but depends heavily on which downstream task you choose and how much training data is available. Bias evaluation using WEAT detects specific pre-defined biases but cannot assess all forms of representational harm.

Coverage is a neglected dimension. Two embedding sets might have similar intrinsic scores while differing greatly in vocabulary size. The smaller vocabulary might score higher simply because its words are more common and easier to represent well. Reporting coverage alongside quality metrics is essential for fair comparisons, especially when comparing embeddings trained on different corpora or with different vocabulary construction strategies.

Visualization is both the most informative and most misleading evaluation tool. t-SNE and UMAP projections are excellent for qualitative exploration and debugging, but they should never be used as quantitative evidence. Cluster separability in a 2D projection depends heavily on projection parameters (especially t-SNE's perplexity), the random initialization, and which words you choose to include. It's easy to make embeddings look impressive or poor depending on how you configure the visualization. Any visualization should clearly report its parameters so readers can assess what they're seeing.

Bias evaluation deserves to be a standard part of any embedding assessment, especially for embeddings used in user-facing systems. WEAT provides a starting point, but it tests specific pre-defined associations. Real-world bias is multidimensional and often implicit, operating through indirect associations that WEAT doesn't test. Running WEAT and finding small effect sizes does not mean the embeddings are bias-free; it means they don't exhibit the specific biases you explicitly tested for.

The field has also begun questioning whether static word embeddings should be evaluated this way at all. Static embeddings give each word one vector regardless of context, which is fundamentally at odds with the context-dependence of meaning. Evaluation on word similarity pairs, which present words out of context, may be measuring something that doesn't quite exist: a word's context-free meaning. This is one of the motivations for contextualized representations, which we'll explore in the next part.

This chapter covered the major methods for evaluating word embeddings, building from simple pairwise similarity measures to multi-dimensional evaluation protocols.

Intrinsic evaluation tests embeddings directly using:

• Word similarity (SimLex-999, WordSim-353, MEN, RG-65, SimVerb-3500), measured by Spearman rank correlation between embedding cosine similarities and human judgments. SimLex-999 is preferred over WordSim-353 because it carefully distinguishes semantic similarity from associative relatedness.
• Analogy accuracy (Google Analogy Dataset), measured by 3CosAdd accuracy. Performance on semantic versus syntactic subcategories reveals what kinds of relationships the model has learned.

Visualization projects embeddings to 2D using t-SNE or UMAP for qualitative inspection. t-SNE excels at revealing local cluster structure; UMAP better preserves global relationships and is faster at scale. Both should be treated as qualitative tools, not quantitative evidence.

Extrinsic evaluation measures downstream task performance (text classification, NER, POS tagging, STS, QA) with embeddings frozen to isolate their contribution. This is the most reliable signal for practical deployment decisions, but it requires building and evaluating a full downstream system.

Bias evaluation uses WEAT to measure differential associations between target word groups and attribute word groups, detecting societal biases encoded in the embedding space. Debiasing approaches include hard debiasing, data augmentation, and counterfactual training.

Key pitfalls include conflating similarity with relatedness, inflated scores from high OOV rates, the hubness problem in high-dimensional cosine similarity, the weak and unreliable correlation between intrinsic and extrinsic scores, test set contamination, and reporting without statistical significance.

The right evaluation strategy depends on your use case. During development, fast intrinsic evaluations guide architectural choices. Before deployment, extrinsic evaluation on your actual downstream task is the most reliable signal. For any production system, bias evaluation is not optional. And in all cases, report your methodology, coverage rates, and baselines alongside the metrics themselves.

Part XXIV: BERT and Variants moves beyond static word embeddings to contextualized representations, where the same word receives different vectors depending on its surrounding context. The evaluation challenges become even richer when a word no longer lives at a fixed point in space but occupies different positions depending on what it means in a given sentence.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about embedding evaluation.