Embedding Evaluation › Intrinsic vs Extrinsic Evaluation
Language AI HandbookAll chapters Embedding Evaluation › Intrinsic vs Extrinsic EvaluationOne section · Sections in this chapter2 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The distinction between intrinsic and extrinsic evaluation is fundamental and shapes every evaluation decision you'll make. Understanding why each exists and what trade-offs it embodies will help you choose the right approach for any given situation.
Intrinsic evaluation tests the embeddings directly on proxy tasks designed to measure semantic quality. The logic is: if embeddings capture meaning well, words with similar meanings should have similar vectors. You can measure this by comparing cosine similarities to human judgments, or by checking whether vector arithmetic recovers known relationships. Intrinsic evaluation answers the question "do these embeddings encode meaning in a principled way?" without requiring you to build and train an entire NLP system.
Extrinsic evaluation plugs embeddings into a real NLP system and measures task performance. The logic is: embeddings are tools for downstream applications, so their quality should ultimately be judged by whether they help those applications work better. Extrinsic evaluation answers the question "do these embeddings make my system better?"
Intrinsic evaluation assesses embedding quality directly using proxy tasks like word similarity correlation or analogy accuracy.
Extrinsic evaluation assesses embedding quality indirectly by measuring performance on a downstream NLP task like text classification or named entity recognition.
The trade-off between them is real and consequential. Intrinsic evaluation is fast, cheap, and interpretable. You can benchmark dozens of embedding models in minutes without building a downstream system. The results are immediately meaningful: a Spearman correlation of 0.75 on SimLex-999 tells you something concrete about whether the model assigns high similarity to words humans find similar.
The problem is that intrinsic evaluation can mislead you. Embeddings that score well on word similarity might still underperform on downstream tasks if the similarity axes they capture don't align with what the task needs. A model that excels at semantic similarity might be mediocre for syntactic tasks. A model with excellent analogy accuracy on capital-country pairs might still fail at morphological tasks. The proxy tasks are, at the end of the day, proxies.
Extrinsic evaluation is the ground truth. If your embeddings improve accuracy on your NER system, they're better for that task, regardless of what any similarity benchmark says. But the cost is significant: you need to build, train, and evaluate a complete downstream system, which can take hours or days. You also face the confound that downstream performance reflects both the embeddings and the downstream model architecture, training data, and hyperparameters.
Most practitioners use both: intrinsic evaluation to guide quick development iterations and extrinsic evaluation to validate before deployment. Think of intrinsic evaluation as a fast filter and extrinsic evaluation as the final judge.