Embedding Evaluation › Downstream Task Evaluation
Language AI HandbookAll chapters Embedding Evaluation › Downstream Task EvaluationOne section · Sections in this chapter6 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Intrinsic evaluation tells you about embedding properties. Extrinsic evaluation tells you whether those properties help with NLP tasks. The standard approach is to use embeddings as features in a downstream model and measure task performance. This is where the rubber meets the road: the only question that ultimately matters is whether better embeddings lead to better systems.
Common Downstream Tasks
The most frequently used extrinsic benchmarks are:
- Text classification: Sentiment analysis (positive/negative/neutral) and topic classification. Usually measured with accuracy or macro-averaged F1 across classes.
- Named entity recognition (NER): Identifying spans of text that refer to named entities like people, organizations, and locations. Measured with entity-level F1.
- Part-of-speech tagging: Labeling each word with its grammatical role (noun, verb, adjective, etc.). Measured with token-level accuracy.
- Semantic textual similarity (STS): Predicting how similar two sentences are, scored on a continuous scale. Measured with Pearson or Spearman correlation against human judgments, analogous to word similarity evaluation.
- Question answering: Locating or generating answers to questions given a context passage. Measured with exact match and token-overlap F1.
The choice of downstream task matters enormously for what you learn about the embeddings. Sentiment analysis is dominated by word-level semantics: knowing that "terrible" is near "awful" in the embedding space is directly useful. NER is dominated by morphological and distributional patterns: words that appear in similar syntactic positions (company names, person names) should be near each other. POS tagging is heavily syntactic: embeddings that capture grammatical function well will outperform semantically rich embeddings on this task. If you evaluate only on sentiment, you'll miss systematic weaknesses that matter for other applications.
Probing vs Fine-tuning
Two protocols exist for extrinsic evaluation, and choosing between them significantly affects what the results tell you.
Frozen embeddings: Use the embeddings as fixed input features, and train only the downstream model on top of them. The embeddings themselves are never updated. This directly measures the quality of the embedding space because the downstream model cannot compensate for embedding deficiencies. If the embeddings don't encode the information the downstream task needs, no amount of clever architecture above them can recover it.
Fine-tuning: Initialize with pretrained embeddings but allow them to update during downstream training. The final performance reflects both the initial quality of the embeddings and the downstream data. Fine-tuned models almost always outperform frozen-embedding models, because fine-tuning allows the model to specialize the representations for the specific task. But this makes it harder to compare embedding models: if both achieve high downstream accuracy after fine-tuning, it may mean the downstream data was sufficient to overcome any initial embedding differences, not that the embeddings were equally good.
For pure embedding evaluation, frozen embeddings are more informative. Fine-tuning is more appropriate when you want to know the practical performance ceiling achievable with a given embedding model as initialization.
Let's implement a simple downstream evaluation using text classification:
import numpy as np
np.random.seed(42)
# Simulate a simple sentence embedding task
# Each "document" is a bag of words; embedding = mean of word embeddings
def embed_document(text, embeddings, dim=50):
"""Embed a document as the mean of its word embeddings."""
tokens = text.lower().split()
vecs = [embeddings[t] for t in tokens if t in embeddings]
if not vecs:
return np.zeros(dim)
return np.mean(vecs, axis=0)
# Create a synthetic sentiment dataset
positive_docs = [
"the computer software works great",
"my laptop program runs fast",
"the phone app is amazing",
"great software runs smoothly",
"laptop computer program excellent",
]
negative_docs = [
"the car broke down again",
"truck bus delay terrible",
"automobile engine failure bad",
"bus arrived very late",
"car truck broke terrible",
]
all_docs = positive_docs + negative_docs
labels = [1] * len(positive_docs) + [0] * len(negative_docs)
# Embed all documents
doc_embeddings = np.array([embed_document(doc, embeddings) for doc in all_docs])from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
# Train a logistic regression classifier on top of embeddings
# Use cross-validation for evaluation
clf = LogisticRegression(max_iter=1000, random_state=42)
cv_scores = cross_val_score(
clf, doc_embeddings, labels, cv=5, scoring="accuracy"
)Downstream Evaluation: Text Classification
==================================================
Dataset: 10 documents, 2 classes
Embedding dimension: 50
5-fold Cross-Validation Results:
Fold 1: 1.000
Fold 2: 1.000
Fold 3: 1.000
Fold 4: 1.000
Fold 5: 1.000
Mean accuracy: 1.000 ± 0.000
Training set classification report:
precision recall f1-score support
Negative 1.00 1.00 1.00 5
Positive 1.00 1.00 1.00 5
accuracy 1.00 10
macro avg 1.00 1.00 1.00 10
weighted avg 1.00 1.00 1.00 10This pipeline shows the core extrinsic evaluation workflow. In practice, you'd replace these synthetic documents with a real labeled dataset (SST-2 for sentiment, CoNLL-2003 for NER, etc.) and compare multiple embedding models side by side on the same task.
Mean Pooling as a Document Representation
The approach in the code above, averaging word embeddings to represent a document, is called mean pooling. It's a simple and surprisingly effective baseline. Each word contributes its embedding to the average, so the document vector reflects the average semantic content of its words. Words that appear multiple times contribute proportionally more if you use a plain average, or equally if you use a unique-word average.
Mean pooling has known limitations. It ignores word order completely: "dog bites man" and "man bites dog" produce identical document embeddings. It also treats all words equally: function words like "the" and "of" contribute as much as content words, which can dilute the semantic signal. More sophisticated alternatives like weighted averaging (down-weighting common words using IDF), concatenating min/max/mean pools, or training a full sequence model on top of the embeddings address these limitations at the cost of additional complexity.
Comparing Embedding Models
The true value of extrinsic evaluation is comparison. Running multiple embedding models through the same downstream pipeline produces directly comparable performance numbers:
def simulate_model_comparison():
"""
Simulate comparing multiple embedding models on the same downstream task.
Returns accuracy scores for each model.
"""
np.random.seed(123)
# Simulate different embedding qualities
model_configs = [
("Random Baseline", 0.50, 0.08),
("Word2Vec (small)", 0.68, 0.05),
("Word2Vec (large)", 0.75, 0.04),
("GloVe 50d", 0.72, 0.04),
("GloVe 300d", 0.80, 0.03),
("FastText", 0.78, 0.04),
("FastText + Subwords", 0.82, 0.03),
]
results = []
for name, mean_acc, std_acc in model_configs:
# Simulate 5-fold CV scores around the mean
scores = np.random.normal(mean_acc, std_acc, 5)
scores = np.clip(scores, 0.0, 1.0)
results.append(
{
"model": name,
"mean": scores.mean(),
"std": scores.std(),
"scores": scores,
}
)
return results
comparison_results = simulate_model_comparison()
The comparison makes trade-offs immediately visible: larger GloVe embeddings outperform smaller ones, FastText with subword information performs best, and all learned embeddings far exceed random baselines. Error bars from cross-validation are essential here: without them, you can't judge whether observed differences are meaningful or within random variation.