Word Analogy › Analogy Evaluation Datasets

To systematically measure embedding quality, researchers created standardized analogy datasets. These allow consistent comparison across different embedding methods, training corpora, and hyperparameter settings. Performance is measured as the fraction of questions answered correctly using top-1 accuracy: a question counts as correct only if the predicted word exactly matches the expected answer word.

The framing of analogy evaluation as accuracy measurement was consequential for the field. It turned embedding quality from a fuzzy, qualitative judgment into a number on a leaderboard, which focused research effort and accelerated progress. But it also created pressures that we will examine critically in the limitations section.

The most widely used benchmark is the Google Analogy Dataset, introduced by Mikolov et al. (2013) alongside the original Word2Vec paper. It contains 19,544 analogy questions organized into two categories: semantic and syntactic.

Semantic analogies cover real-world, encyclopedic relationships between words:

• Capital-country: Paris:France :: Tokyo:Japan (94 pairs, asking for the capital given the country and one known capital)
• Capital-world: Oslo:Norway :: Brussels:Belgium (506 pairs covering world capitals more broadly)
• City-state: Chicago:Illinois :: Houston:Texas (2,467 pairs for US cities and their states)
• Currency: dollar:USA :: yen:Japan (866 pairs linking currencies to countries)
• Family: boy:girl :: brother:sister (506 pairs covering kinship and gender relationships)

Syntactic analogies cover morphological transformations, testing whether the model has learned grammatical structure:

• Adjective-to-adverb: quick:quickly :: slow:slowly (992 pairs)
• Opposite: ethical:unethical :: responsible:irresponsible (812 pairs)
• Comparative: big:bigger :: fast:faster (1,332 pairs)
• Superlative: big:biggest :: fast:fastest (1,122 pairs)
• Present-participle: go:going :: run:running (1,056 pairs)
• Past-tense: dancing:danced :: swimming:swam (1,560 pairs)
• Plural nouns: mouse:mice :: foot:feet (1,332 pairs)
• Plural verbs: work:works :: speak:speaks (1,002 pairs)

The dataset construction introduces a dependency: the dataset was constructed by taking word pairs and generating all pairwise combinations within each category. If you have the pairs (Paris, France) and (Tokyo, Japan) and (Berlin, Germany), you generate six analogies: Paris:France::Tokyo:Japan, Paris:France::Berlin:Germany, Tokyo:Japan::Berlin:Germany, and so on in both directions. This means the analogy questions within a category are not independent; they share words. This creates correlation in the evaluation that single accuracy numbers don't reflect.

The Microsoft Research (MSR) Syntactic Analogies Dataset focuses exclusively on syntactic relationships, with 8,000 questions covering eight morphological transformations. The dataset was designed to provide a more targeted test of morphological competence than the mixed Google dataset. It is particularly useful when comparing models on grammatical regularities, where differences between embedding methods are often more pronounced because morphological variation has more consistent distributional signal than semantic variation.

The MSR dataset emphasizes regular morphological patterns: plurals, comparatives, past tenses. This makes it a strong test for models trained to capture syntax, and a weaker test for models that prioritize semantic content. In practice, models like FastText, which use character-level subword information, perform better on MSR than purely token-level models do, because they have access to the morphological surface form directly.

The Bigger Analogy Test Set (BATS), introduced by Gladkova et al. (2016), expands coverage to 40 linguistic relation types with 98,000 questions. BATS adds several categories not found in the Google or MSR datasets:

• Encyclopedic relationships: animal-typical-habitat (bird:nest::fish:water), country-language (France:French::Germany:German)
• Lexicographic relationships: synonyms (fast:rapid::cold:frigid) and antonyms (hot:cold::fast:slow)
• Morphological derivation: verb-to-noun (run:runner::swim:swimmer), adjective-to-noun (happy:happiness::sad:sadness)
• Morphological inflection: regular plurals, irregular plurals, past tense, third-person singular

The value of BATS is its diagnostic granularity. Rather than reporting a single accuracy number, you get 40 accuracy scores, one per relation type. This reveals which relationship categories a given embedding captures well and which it misses. A model might score 90% on capital-country but only 40% on synonyms, indicating that it has learned encyclopedic facts but not lexical semantic relationships. This profile is far more informative than a single aggregate number.