Embedding Evaluation › Word Embedding Evaluation: Intrinsic and Extrinsic Methods
Language AI HandbookAll chapters Embedding Evaluation › Word Embedding Evaluation: Intrinsic and Extrinsic MethodsOne section · Sections in this chapter1 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
You've trained a Word2Vec model, loaded pre-trained GloVe vectors, or fine-tuned FastText embeddings. Now comes the critical question: are these embeddings any good? And what does "good" even mean for a vector representation of a word?
This is harder than it sounds. Word embeddings are dense, high-dimensional vectors with no obvious ground truth. You can't look at the number [0.32, -0.15, 0.78, ...] and judge whether it's a good representation of "apple." You need evaluation methods that translate embedding quality into measurable quantities that correlate with real usefulness.
The challenge runs even deeper. Word embeddings are general-purpose representations intended to work across many downstream applications. A set of embeddings that excels at capturing semantic similarity between nouns might perform poorly on syntactic tasks like part-of-speech tagging, and vice versa. Any single evaluation method necessarily captures only a slice of what makes an embedding useful. This means you need a portfolio of evaluation approaches rather than a single definitive score.
Embedding evaluation has two distinct flavors. Intrinsic evaluation measures properties of the embeddings themselves: do similar words cluster together? Do vector arithmetic operations capture semantic relationships? These are fast, interpretable tests that give immediate feedback during development. Extrinsic evaluation measures the performance of a downstream NLP system that uses the embeddings: do they improve named entity recognition, sentiment analysis, or machine translation? This is the gold standard but expensive to compute.
This chapter covers both approaches in depth. We'll work through the major word similarity datasets, understand why Spearman correlation is the right metric, implement analogy evaluation and examine the 3CosAdd method in detail, visualize embeddings with t-SNE and UMAP, run a downstream evaluation, and examine the critical issue of embedding bias. Along the way, we'll highlight common pitfalls that lead researchers to draw incorrect conclusions from evaluation results.