Embedding Evaluation › Summary

This chapter covered the major methods for evaluating word embeddings, building from simple pairwise similarity measures to multi-dimensional evaluation protocols.

Intrinsic evaluation tests embeddings directly using:

• Word similarity (SimLex-999, WordSim-353, MEN, RG-65, SimVerb-3500), measured by Spearman rank correlation between embedding cosine similarities and human judgments. SimLex-999 is preferred over WordSim-353 because it carefully distinguishes semantic similarity from associative relatedness.
• Analogy accuracy (Google Analogy Dataset), measured by 3CosAdd accuracy. Performance on semantic versus syntactic subcategories reveals what kinds of relationships the model has learned.

Visualization projects embeddings to 2D using t-SNE or UMAP for qualitative inspection. t-SNE excels at revealing local cluster structure; UMAP better preserves global relationships and is faster at scale. Both should be treated as qualitative tools, not quantitative evidence.

Extrinsic evaluation measures downstream task performance (text classification, NER, POS tagging, STS, QA) with embeddings frozen to isolate their contribution. This is the most reliable signal for practical deployment decisions, but it requires building and evaluating a full downstream system.

Bias evaluation uses WEAT to measure differential associations between target word groups and attribute word groups, detecting societal biases encoded in the embedding space. Debiasing approaches include hard debiasing, data augmentation, and counterfactual training.

Key pitfalls include conflating similarity with relatedness, inflated scores from high OOV rates, the hubness problem in high-dimensional cosine similarity, the weak and unreliable correlation between intrinsic and extrinsic scores, test set contamination, and reporting without statistical significance.

The right evaluation strategy depends on your use case. During development, fast intrinsic evaluations guide architectural choices. Before deployment, extrinsic evaluation on your actual downstream task is the most reliable signal. For any production system, bias evaluation is not optional. And in all cases, report your methodology, coverage rates, and baselines alongside the metrics themselves.

Part XXIV: BERT and Variants moves beyond static word embeddings to contextualized representations, where the same word receives different vectors depending on its surrounding context. The evaluation challenges become even richer when a word no longer lives at a fixed point in space but occupies different positions depending on what it means in a given sentence.