Word Analogy › Analogy Accuracy as a Metric
Language AI HandbookAll chapters Word Analogy › Analogy Accuracy as a MetricOne section · Sections in this chapter6 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The standard accuracy metric counts how often the correct word is the top-ranked answer:
An important implementation detail: questions where any of the four words fall outside the vocabulary are typically skipped rather than counted as errors. This means accuracy is measured only over the questions the model can attempt, not all questions in the dataset. This can inflate apparent accuracy for models with small vocabularies, since the hardest questions (involving rare words) are silently dropped.
Typical performance benchmarks on the Google dataset have shifted over the years as training data and model architectures have improved:
- Word2Vec (skip-gram, 300d, trained on Google News, 100 billion tokens): approximately 65-70% overall accuracy
- GloVe (300d, trained on Common Crawl, 840 billion tokens): approximately 75-80% overall accuracy
- FastText (300d, trained on Common Crawl): approximately 75-80% overall accuracy, with higher syntactic accuracy
These numbers vary considerably with vocabulary size, training corpus size and quality, embedding dimension, training objective, and random initialization. They should be treated as rough benchmarks for understanding the relative capabilities of different approaches, not as definitive comparisons. A fair comparison requires controlling all other variables except the factor you are studying.
def evaluate_analogies(analogy_list, embeddings):
"""
Evaluate a list of analogies using 3CosAdd.
analogy_list: list of (a, b, c, correct_d) tuples
Returns accuracy as a fraction.
"""
correct = 0
total = 0
results_detail = []
for a, b, c, correct_d in analogy_list:
# Skip if any word is not in vocabulary
if any(w not in embeddings for w in [a, b, c, correct_d]):
continue
ranked = three_cos_add(a, b, c, embeddings)
predicted = ranked[0][0]
is_correct = predicted == correct_d
correct += is_correct
total += 1
results_detail.append(
{
"analogy": f"{a}:{b} :: {c}:?",
"correct": correct_d,
"predicted": predicted,
"hit": is_correct,
}
)
accuracy = correct / total if total > 0 else 0.0
return accuracy, results_detail
# Synthetic analogy dataset using our 2D embeddings
analogies = [
("man", "woman", "king", "queen"),
("man", "woman", "prince", "princess"),
("man", "woman", "uncle", "aunt"),
("king", "man", "queen", "woman"),
("prince", "man", "princess", "woman"),
("uncle", "man", "aunt", "woman"),
]
accuracy, detail = evaluate_analogies(analogies, embeddings_2d)Analogy Evaluation Results ============================================================ Total questions: 6 Correct: 6 Accuracy: 100.0% Analogy Expected Predicted Correct ------------------------------------------------------------ man:woman :: king:? queen queen YES man:woman :: prince:? princess princess YES man:woman :: uncle:? aunt aunt YES king:man :: queen:? woman woman YES prince:man :: princess:? woman woman YES uncle:man :: aunt:? woman woman YES
Our synthetic embeddings achieve perfect accuracy because the gender offset is geometrically exact. Real embeddings show imperfect accuracy, with errors most common on ambiguous words, irregular forms, or relationship types where the training data does not provide consistent distributional signal.
Accuracy by Relationship Category
Not all relationship types are equally easy for analogy completion. The variation across categories is instructive because it reveals which aspects of language are captured as consistent linear structure and which are not.
Geographic relationships typically score the highest. Country capitals are consistently mentioned together in the same kinds of documents: reference texts, travel articles, news reports. The co-occurrence patterns for France and Paris, Germany and Berlin, and Japan and Tokyo are structurally similar, creating a consistent offset direction in the embedding space. When the training corpus is large enough to have seen hundreds of country-capital mentions, the model reliably learns this relationship.
Family relationships score high for similar reasons: "brother" and "sister" appear in analogous sentence contexts, and the gendered role opposition is strong and consistent across many sentences. The offset between brother and sister is parallel to the offset between uncle and aunt, husband and wife, king and queen.
Morphological relationships show more variation. Regular morphological transformations (quick:quickly, big:bigger) score well because the pattern is systematic and appears thousands of times in natural text. Irregular forms (go:went, good:better) score lower because the surface forms don't share a consistent directional relationship: "went" doesn't look like "go," so their embeddings don't share the same geometric displacement as "walk" and "walked."
The variation across categories reflects the underlying distributional structure of natural language. Geographic relationships score high because country capitals appear consistently in structured reference text. Currency pairs score lower because many currency names are polysemous or appear in fewer training contexts. Irregular past tenses like "go:went" score lower than regular ones like "dance:danced" because the irregular forms don't share the same directional offset: the geometric path from "go" to "went" is not parallel to the path from "dance" to "danced."