Word Analogy: Vector Arithmetic and Semantic Relationships

Something remarkable happens when you train Skip-gram or CBOW on billions of words: the resulting embeddings exhibit structured geometric relationships that mirror semantic ones. The famous example is $\vec{king} - \vec{man} + \vec{woman} \approx \vec{queen}$. This isn't a coincidence or a numerical trick. The vector from "man" to "woman" captures a gender relationship, and adding it to "king" lands near "queen." The embedding space has learned that kingship relates to queenship the same way manhood relates to womanhood. Royalty and gender are encoded as separate, compositional directions in the same high-dimensional space.

This phenomenon, known as word analogy or vector arithmetic, became one of the most celebrated results in NLP when Mikolov et al. introduced it in 2013 alongside Word2Vec. It was striking evidence that neural embeddings do more than memorize co-occurrence statistics. They organize knowledge into a geometric structure where relationships can be expressed as arithmetic operations. You can reason about language by adding and subtracting vectors.

This chapter explores exactly how and why this works. We develop the parallelogram model that makes analogy completion possible, derive the 3CosAdd and 3CosMul methods for finding answers, examine the standard evaluation datasets researchers use to compare embedding quality, and take an honest look at what analogy performance reveals and where it misleads. By the end, you will understand how to run analogy evaluations, why the technique matters, when it succeeds, and when it fails in ways that matter for real applications.

Word analogies rest on a geometric intuition: if two word pairs share the same semantic relationship, the vectors connecting them should be parallel and of similar length. This is the parallelogram model, and it is the foundation for all analogy-based evaluation.

Consider the relationship "capital city of." Paris is the capital of France, Tokyo is the capital of Japan, and Berlin is the capital of Germany. If embeddings capture this relationship consistently, then the vector pointing from France to Paris should be roughly the same as the vector from Japan to Tokyo and from Germany to Berlin. Each of those difference vectors encodes the same directional displacement: "country to its capital city." This directional consistency is what makes analogy completion possible. If you know three of the four words in a pair of analogous relationships, you can predict the fourth by applying the relationship vector.

To see why this structure would arise during training, think about how Skip-gram learns. Words that appear in similar contexts end up with similar embeddings. France, Germany, and Japan all appear in contexts like "the prime minister of ___", "the economy of ___", "traveled to ". Paris, Tokyo, and Berlin appear in contexts like "flew into ___ this morning", " is a vibrant city", "the Eiffel Tower in ___". The contextual patterns for all three countries are similar to each other, and the contextual patterns for all three capitals are similar to each other. The model pulls France, Germany, and Japan into a cluster, and Paris, Tokyo, and Berlin into another cluster. The geometric displacement from one cluster to the other encodes the capital-of relationship. When this happens for dozens of country-capital pairs, the displacement vectors become aligned.

The model is called a parallelogram because of the shape formed when you draw the four word vectors as points in space and connect them with the relationship arrows. The two "relationship" edges are parallel: $\vec{man} \to \vec{woman}$ runs parallel to $\vec{king} \to \vec{queen}$. The two "word class" edges are also parallel: $\vec{man} \to \vec{king}$ runs parallel to $\vec{woman} \to \vec{queen}$. This is the geometric definition of a parallelogram, and it is the structure we are detecting when we evaluate analogies.

The parallelogram structure shown above is idealized, with a perfectly consistent gender offset. Real embeddings have noise: the offsets aren't perfectly parallel because each word is trained from noisy co-occurrence statistics in natural text. "King" appears in contexts that "prince" does not, and "man" appears in everyday contexts that "uncle" does not. So the offsets won't be identical. But when embeddings are trained well on large corpora, the relationship vectors are similar enough in direction that analogy completion works reliably for many relationship types. The key insight is that it doesn't need to be perfect. You just need the correct answer to be closer to the target than any other word in the vocabulary.

It is worth pausing to understand why this linear structure should emerge at all from a model that is simply trained to predict context words. The answer connects distributional semantics to linear algebra in a way that illuminates why the analogy trick works.

When Skip-gram trains on a sentence like "The king ruled his kingdom wisely," it updates the embedding for "king" to better predict the surrounding words: "the," "ruled," "his," "kingdom," "wisely." The same word "ruled" also appears around other powerful entities: generals, prime ministers, emperors. So "general," "king," and "emperor" end up with overlapping context distributions. They get pulled toward similar embedding regions.

Now consider gender. Every word that participates in a gender-paired relationship (man/woman, king/queen, actor/actress, waiter/waitress) appears in two kinds of contexts: its own semantic contexts (royalty, service, performance) and gender-indicative contexts. "Queen" appears with "she," "her," "her majesty." "King" appears with "he," "his," "his majesty." The model learns a representation for "queen" that is pulled toward the "royalty" region (because of shared semantic contexts with "king") and simultaneously displaced in a direction associated with feminine pronouns and references (because of gender-indicative context words). This displacement is approximately the same for all gender pairs, because they all get pushed in the same direction by the same set of gendered context words.

The result is a shared directional offset: a gender axis in the embedding space. The position along this axis distinguishes male from female terms; the position in the orthogonal subspace distinguishes the semantic category (royal vs. occupational vs. familial). This decomposition of meaning into compositional subspaces is the deep reason analogy arithmetic works.

The classic analogy task asks: "$a$ is to $b$ as $c$ is to ?". If the relationship between $a$ and $b$ is the same as the relationship between $c$ and $d$, then the offset vectors should be approximately equal: $\vec{b} - \vec{a} \approx \vec{d} - \vec{c}$. Rearranging this equation gives us the target vector we are searching for:

where:

• $\vec{a}$, $\vec{b}$, $\vec{c}$, $\vec{d}$: the embedding vectors for the respective words
• $\vec{b} - \vec{a}$: the relationship vector capturing how $b$ differs from $a$
• $\vec{c}$: the starting point from which we apply the relationship

The process has three conceptually distinct steps. First, compute the relationship vector $\vec{b} - \vec{a}$, which captures the directional change associated with the relationship. For the gender analogy, this is the vector pointing from "man" to "woman," representing the direction of femininity in embedding space. Second, apply this offset to $\vec{c}$ to predict where $\vec{d}$ should be. Adding the gender offset to "king" produces a point that should be near the female-equivalent-of-king, which is "queen." Third, search the vocabulary for the word whose embedding is closest to this computed target vector. The word with the nearest embedding is the predicted answer.

The simplicity of this process is remarkable. Three vector additions and a nearest-neighbor search are enough to solve thousands of semantic and syntactic analogy questions.

The distance of zero confirms that our synthetic embeddings exhibit perfect parallelogram structure: the computed target vector exactly matches "queen." In real embeddings there is noise, so we must search the vocabulary for the nearest word by some distance metric, which brings us to the question of which metric to use.

We have established the intuition: semantic relationships create parallel vectors, and we can complete analogies by adding relationship offsets. But how do we find the answer word across tens of thousands of vocabulary items? The target vector we compute is a point in high-dimensional space. We need to find the vocabulary word whose embedding lies closest to that point, and we need a distance measure that is appropriate for the geometry of embedding spaces.

The standard approach searches for the vocabulary word $d$ that maximizes cosine similarity to the target vector $\vec{b} - \vec{a} + \vec{c}$:

where:

• $V$: the full vocabulary
• $\{a, b, c\}$: excluded because we want a fourth, distinct word as the answer
• $\cos(\vec{x}, \vec{y})$: the cosine similarity between vectors $\vec{x}$ and $\vec{y}$

This formulation is called 3CosAdd because it uses three vectors summed additively before computing cosine similarity. The name also reflects how it is equivalently written as a sum of three individual cosine similarities, which we'll derive below.

You might wonder why we use cosine similarity rather than simply finding the word with minimum Euclidean distance to the target vector. The problem is that word embeddings have different magnitudes. Frequent words like "the," "a," and "is" appear in enormous numbers of training contexts and tend to receive large embedding norms because their gradient updates accumulate over many training examples. Rare words like "czar" or "serration" appear in few contexts and have smaller norms. Euclidean distance conflates directional alignment with magnitude matching: it would favor words whose norms happen to be similar to the target's magnitude, even if they point in a completely different direction.

Cosine similarity solves this by focusing purely on the angle between vectors, ignoring their lengths entirely. It measures the cosine of the angle between two vectors:

where:

• $\vec{x} \cdot \vec{y}$: the dot product of the vectors, computed as the sum of element-wise products $\sum_i x_i y_i$
• $\|\vec{x}\|$, $\|\vec{y}\|$: the Euclidean norms (lengths) of the vectors, computed as $\sqrt{\sum_i x_i^2}$ and $\sqrt{\sum_i y_i^2}$

The denominator normalizes by both vector lengths, making the result range from -1 to 1 regardless of magnitude. Two vectors pointing in exactly the same direction have cosine similarity 1. Perpendicular vectors have cosine similarity 0. Opposite-pointing vectors yield -1. For analogy completion, we care about which word's embedding points the same direction as our target, not which word has a similar magnitude.

There is one subtlety. When we compute $\vec{b} - \vec{a} + \vec{c}$, the resulting target vector is not normalized. It is a raw algebraic combination of three embedding vectors. Cosine similarity then compares this unnormalized target against each candidate in the vocabulary. The comparison is still direction-based, which is what we want, because cosine similarity normalizes both arguments before measuring the angle.

Queen ranks first because the synthetic embeddings were constructed with perfect parallelogram structure. In real embeddings trained on text, the correct answer typically ranks first or second for well-studied relationship types, with errors most common on ambiguous, rare, or morphologically irregular words.

3CosAdd can be rewritten in an equivalent form that makes its logic more transparent. Rather than computing cosine similarity to a single composite target vector, 3CosAdd turns out to maximize a combination of three individual cosine similarities. This equivalence reveals exactly what the algorithm is selecting for.

The derivation begins with the observation that cosine similarity is linear in its first argument when the second argument is fixed. Expanding $\cos(\vec{d},\ \vec{b} - \vec{a} + \vec{c})$ and exploiting the bilinearity of the dot product yields the equivalent form:

where:

• $\cos(\vec{d}, \vec{b})$: reward words similar to $b$, the "answer template" providing the target semantic role
• $-\cos(\vec{d}, \vec{a})$: penalize words similar to $a$, subtracting the source concept
• $\cos(\vec{d}, \vec{c})$: reward words similar to $c$, the query word we are completing

This reformulation reveals the algorithm's selection criterion more clearly than the vector addition form. You want a word that resembles $b$ (in the same role as the target), resembles $c$ (related to the query), but does not resemble $a$ (not the same as the source). For "man is to woman as king is to ?", you want a word that is like "woman" (feminine), like "king" (royal), but not like "man" (masculine). "Queen" satisfies all three conditions more strongly than any other word in the vocabulary.

The subtraction term determines the result. Without it, you would just find the word most similar to both "woman" and "king", which might be "ruler" or "monarch" or some other word that is royal and somewhat gender-associated. The subtraction penalizes "man" and similar male terms, forcing the algorithm to find a word that has the femininity of "woman" without having the maleness of "man."

Note that this equivalence is approximate. It holds exactly only when all vectors are unit-normalized, because the derivation assumes the cosine similarity of the sum equals the sum of cosine similarities. In practice, Word2Vec and GloVe embeddings are not perfectly normalized, so the two forms can give slightly different rankings. The explicit three-cosine form is sometimes preferred in practice because it can be more numerically stable.

Both formulations yield identical rankings on these normalized 2D embeddings, confirming they are mathematically equivalent in this case. When working with high-dimensional, unnormalized embeddings, the two forms can differ slightly. In practice, you can use either; the explicit form is often clearer when implementing from scratch and the vector addition form is faster for large vocabulary searches.

While 3CosAdd is the standard approach, it has a known vulnerability related to the subtraction operation. The term $-\cos(\vec{d}, \vec{a})$ penalizes words that are similar to $a$, but when cosine similarity values are small and near zero, the arithmetic can amplify noise in unpredictable ways. A word that is slightly negatively correlated with $a$ for unrelated reasons will receive a spurious bonus. Conversely, a word that is the correct answer but happens to have mild positive similarity to $a$ will be penalized unfairly.

Levy, Goldberg, and Dagan (2014) proposed 3CosMul as an alternative that multiplies similarities rather than adding and subtracting them. The intuition is that multiplication is less sensitive to near-zero values: a word with low similarity to anything will score low overall, rather than getting an arbitrary bonus from a negative subtraction.

where:

• $\epsilon$: a small constant (typically 0.001) to prevent division by zero when the denominator is near zero
• $\cos(\vec{d}, \vec{b}) \cdot \cos(\vec{d}, \vec{c})$: the numerator rewards words with high joint similarity to both $b$ and $c$. Multiplication means both conditions must be satisfied simultaneously; a word that is highly similar to $b$ but unrelated to $c$ will score poorly.
• $\cos(\vec{d}, \vec{a}) + \epsilon$: the denominator penalizes similarity to $a$. Words very similar to $a$ are divided out, reducing their score.

The multiplicative structure has an important consequence: the formula requires all cosine similarity values to be non-negative. Cosine similarity ranges from -1 to 1, and negative values in the numerator or denominator would produce undefined or sign-flipped results. The standard implementation shifts all cosine values to the $[0, 1]$ range before applying the formula, using the transformation $\cos' = (\cos + 1) / 2$.

This shift means 3CosMul and 3CosAdd are not directly comparable on the raw scale: they use different transformations of the same underlying similarity values. But both aim to rank the correct answer highest, and empirically both often succeed.

The performance difference between 3CosAdd and 3CosMul is subtle and dataset-dependent. Levy et al. (2014) found that 3CosMul performs better on syntactic analogies such as plurals and verb tenses, while 3CosAdd is competitive on semantic analogies such as capitals and currency pairs. The practical recommendation is to try both when evaluating embeddings on a new dataset and to report whichever performs better (or both).

The intuitive explanation for 3CosMul's advantage on syntactic analogies relates to the noise properties of those relationships. Syntactic patterns are encoded through large numbers of morphologically similar examples. The word "dogs" differs from "dog" in ways that affect many contextual words: it appears with plural verbs, with plural determiners, and in contexts where "dog" appears with singular forms. This creates many weak cosine similarity signals in many directions. The additive method's subtraction can amplify these weak signals into noise. The multiplicative method's requirement for joint similarity to both the target and the query word handles these diffuse signals more reliably.

Both methods correctly identify "queen" as the top-ranked answer. The key difference is visible in the score distributions: 3CosAdd produces signed scores where some candidates fall below zero, while 3CosMul keeps all candidates positive. The gap between the top answer and the rest also differs: the multiplicative formulation tends to create sharper distinctions when the correct answer has strong joint similarity to both $b$ and $c$.

To systematically measure embedding quality, researchers created standardized analogy datasets. These allow consistent comparison across different embedding methods, training corpora, and hyperparameter settings. Performance is measured as the fraction of questions answered correctly using top-1 accuracy: a question counts as correct only if the predicted word exactly matches the expected answer word.

The framing of analogy evaluation as accuracy measurement was consequential for the field. It turned embedding quality from a fuzzy, qualitative judgment into a number on a leaderboard, which focused research effort and accelerated progress. But it also created pressures that we will examine critically in the limitations section.

The most widely used benchmark is the Google Analogy Dataset, introduced by Mikolov et al. (2013) alongside the original Word2Vec paper. It contains 19,544 analogy questions organized into two categories: semantic and syntactic.

Semantic analogies cover real-world, encyclopedic relationships between words:

• Capital-country: Paris:France :: Tokyo:Japan (94 pairs, asking for the capital given the country and one known capital)
• Capital-world: Oslo:Norway :: Brussels:Belgium (506 pairs covering world capitals more broadly)
• City-state: Chicago:Illinois :: Houston:Texas (2,467 pairs for US cities and their states)
• Currency: dollar:USA :: yen:Japan (866 pairs linking currencies to countries)
• Family: boy:girl :: brother:sister (506 pairs covering kinship and gender relationships)

Syntactic analogies cover morphological transformations, testing whether the model has learned grammatical structure:

• Adjective-to-adverb: quick:quickly :: slow:slowly (992 pairs)
• Opposite: ethical:unethical :: responsible:irresponsible (812 pairs)
• Comparative: big:bigger :: fast:faster (1,332 pairs)
• Superlative: big:biggest :: fast:fastest (1,122 pairs)
• Present-participle: go:going :: run:running (1,056 pairs)
• Past-tense: dancing:danced :: swimming:swam (1,560 pairs)
• Plural nouns: mouse:mice :: foot:feet (1,332 pairs)
• Plural verbs: work:works :: speak:speaks (1,002 pairs)

The dataset construction introduces a dependency: the dataset was constructed by taking word pairs and generating all pairwise combinations within each category. If you have the pairs (Paris, France) and (Tokyo, Japan) and (Berlin, Germany), you generate six analogies: Paris:France::Tokyo:Japan, Paris:France::Berlin:Germany, Tokyo:Japan::Berlin:Germany, and so on in both directions. This means the analogy questions within a category are not independent; they share words. This creates correlation in the evaluation that single accuracy numbers don't reflect.

The Microsoft Research (MSR) Syntactic Analogies Dataset focuses exclusively on syntactic relationships, with 8,000 questions covering eight morphological transformations. The dataset was designed to provide a more targeted test of morphological competence than the mixed Google dataset. It is particularly useful when comparing models on grammatical regularities, where differences between embedding methods are often more pronounced because morphological variation has more consistent distributional signal than semantic variation.

The MSR dataset emphasizes regular morphological patterns: plurals, comparatives, past tenses. This makes it a strong test for models trained to capture syntax, and a weaker test for models that prioritize semantic content. In practice, models like FastText, which use character-level subword information, perform better on MSR than purely token-level models do, because they have access to the morphological surface form directly.

The Bigger Analogy Test Set (BATS), introduced by Gladkova et al. (2016), expands coverage to 40 linguistic relation types with 98,000 questions. BATS adds several categories not found in the Google or MSR datasets:

• Encyclopedic relationships: animal-typical-habitat (bird:nest::fish:water), country-language (France:French::Germany:German)
• Lexicographic relationships: synonyms (fast:rapid::cold:frigid) and antonyms (hot:cold::fast:slow)
• Morphological derivation: verb-to-noun (run:runner::swim:swimmer), adjective-to-noun (happy:happiness::sad:sadness)
• Morphological inflection: regular plurals, irregular plurals, past tense, third-person singular

The value of BATS is its diagnostic granularity. Rather than reporting a single accuracy number, you get 40 accuracy scores, one per relation type. This reveals which relationship categories a given embedding captures well and which it misses. A model might score 90% on capital-country but only 40% on synonyms, indicating that it has learned encyclopedic facts but not lexical semantic relationships. This profile is far more informative than a single aggregate number.

The standard accuracy metric counts how often the correct word is the top-ranked answer:

An important implementation detail: questions where any of the four words fall outside the vocabulary are typically skipped rather than counted as errors. This means accuracy is measured only over the questions the model can attempt, not all questions in the dataset. This can inflate apparent accuracy for models with small vocabularies, since the hardest questions (involving rare words) are silently dropped.

Typical performance benchmarks on the Google dataset have shifted over the years as training data and model architectures have improved:

• Word2Vec (skip-gram, 300d, trained on Google News, 100 billion tokens): approximately 65-70% overall accuracy
• GloVe (300d, trained on Common Crawl, 840 billion tokens): approximately 75-80% overall accuracy
• FastText (300d, trained on Common Crawl): approximately 75-80% overall accuracy, with higher syntactic accuracy

These numbers vary considerably with vocabulary size, training corpus size and quality, embedding dimension, training objective, and random initialization. They should be treated as rough benchmarks for understanding the relative capabilities of different approaches, not as definitive comparisons. A fair comparison requires controlling all other variables except the factor you are studying.

Our synthetic embeddings achieve perfect accuracy because the gender offset is geometrically exact. Real embeddings show imperfect accuracy, with errors most common on ambiguous words, irregular forms, or relationship types where the training data does not provide consistent distributional signal.

Not all relationship types are equally easy for analogy completion. The variation across categories is instructive because it reveals which aspects of language are captured as consistent linear structure and which are not.

Geographic relationships typically score the highest. Country capitals are consistently mentioned together in the same kinds of documents: reference texts, travel articles, news reports. The co-occurrence patterns for France and Paris, Germany and Berlin, and Japan and Tokyo are structurally similar, creating a consistent offset direction in the embedding space. When the training corpus is large enough to have seen hundreds of country-capital mentions, the model reliably learns this relationship.

Family relationships score high for similar reasons: "brother" and "sister" appear in analogous sentence contexts, and the gendered role opposition is strong and consistent across many sentences. The offset between brother and sister is parallel to the offset between uncle and aunt, husband and wife, king and queen.

Morphological relationships show more variation. Regular morphological transformations (quick:quickly, big:bigger) score well because the pattern is systematic and appears thousands of times in natural text. Irregular forms (go:went, good:better) score lower because the surface forms don't share a consistent directional relationship: "went" doesn't look like "go," so their embeddings don't share the same geometric displacement as "walk" and "walked."

The variation across categories reflects the underlying distributional structure of natural language. Geographic relationships score high because country capitals appear consistently in structured reference text. Currency pairs score lower because many currency names are polysemous or appear in fewer training contexts. Irregular past tenses like "go:went" score lower than regular ones like "dance:danced" because the irregular forms don't share the same directional offset: the geometric path from "go" to "went" is not parallel to the path from "dance" to "danced."

Analogy performance tells you something real about the internal structure of an embedding space, but interpreting that information requires care. A high analogy score doesn't mean the embeddings are good for every task, and a low score doesn't mean they're bad. Understanding what analogies reveal and what they miss is essential for using them as a diagnostic tool rather than treating them as ground truth.

When analogies work well, they reveal three things about an embedding space. First, the space has learned linear structure: relationships are encoded as consistent directional offsets rather than arbitrary transformations. This is a nontrivial property. Nothing in the training objective directly optimizes for linear relationship structure; it emerges as a consequence of how distributional information is compressed into fixed-dimensional vectors. Second, the model has captured distributional regularities: words that appear in similar roles across the corpus cluster together and form coherent subspaces. Third, the training data was rich enough to separate conceptually distinct but co-occurring words into different geometric regions.

When analogies fail, the reasons are instructive and point to real limitations. The parallelogram assumption breaks down for irregular morphological forms. "Go:went" has the same semantic relationship as "run:ran," but "went" and "ran" are not located in the same geometric region relative to "go" and "run" because the surface forms are completely different, and surface form similarity influences embedding geometry. The irregular past tense of "go" has entirely different contextual statistics than the regular pattern. Polysemy creates another failure mode: "bank" as a financial institution and "bank" as a river bank share a single embedding vector, so any analogy involving "bank" is ambiguous. The model must somehow encode two unrelated senses in the same point in space, which guarantees that at least one sense will be poorly located for analogies. And some relationships are not linear: the relationship between cause and effect is complex and multidimensional. You cannot always express the cause of anything as a fixed offset from its effect.

The relationship between training corpus size and analogy performance is important to understand. More training data generally helps, but the gain is uneven across relationship types. Capital-country relationships benefit from large corpora because there are only about 200 countries and their co-occurrence patterns are saturated after seeing a few thousand examples. Irregular verb forms benefit from large corpora because the number of irregular verbs is small and each one needs to be seen in many contexts to build a reliable embedding. By contrast, relationships involving rare entities (obscure currencies, small cities) are fundamentally limited by data sparsity regardless of corpus size.

Analogy accuracy has become a standard benchmark, but its limitations are substantial enough to warrant careful interpretation. Using it as the primary measure of embedding quality can mislead research and development in subtle but important ways.

The 3CosAdd evaluation excludes the three query words $\{a, b, c\}$ from the candidate set. This prevents trivially returning one of the input words as the answer. For example, without exclusion, "woman" would be a valid response to "man:woman::king:?" since "woman" is already present as the source of the relationship vector. The exclusion is clearly necessary for a meaningful evaluation.

But the exclusion creates a subtler artifact. In many cases, the correct answer word lies very close to one of the excluded input words in embedding space. "Queen" is close to "king" by virtue of being in the royalty cluster. Without exclusion, "king" might be the nearest neighbor to the target vector. With exclusion, the evaluation forces the algorithm to find the second or third nearest neighbor, which happens to be "queen." This means the algorithm is being evaluated partly on its ability to move away from the input words rather than purely on its ability to find the correct answer by relationship structure. Removing the exclusion significantly reduces apparent accuracy, suggesting that a portion of "correct" answers benefit from this artificial constraint.

Analogy accuracy depends simultaneously on vocabulary coverage, embedding dimension, training corpus size, preprocessing choices, and the evaluation method. Comparing accuracy across models is valid only when these factors are controlled. A 300-dimensional model trained on 100 billion tokens should not be directly compared to a 50-dimensional model trained on 1 billion tokens. The differences in accuracy reflect the training conditions as much as the model architecture.

This confounding is not a minor inconvenience. In the original Word2Vec paper, Mikolov et al. reported analogy accuracy as evidence for the superiority of their model over baselines. But the comparison used different training data sizes, making it impossible to attribute the improvement to the model architecture rather than the larger corpus. Subsequent controlled experiments have found that corpus size, preprocessing, and hyperparameter choices explain a large fraction of the performance differences between Word2Vec, GloVe, and other methods.

The deeper problem is that analogy accuracy measures a property of embedding space geometry, not performance on downstream tasks. A model might achieve high analogy accuracy while performing poorly on text classification, named entity recognition, or question answering. The connection between analogy accuracy and task performance is empirically weak.

Faruqui et al. (2016) systematically examined the correlation between intrinsic embedding evaluations, including analogy accuracy and word similarity judgments, and performance on downstream tasks. They found low and inconsistent correlations across tasks and datasets. For some tasks, higher intrinsic scores were associated with slightly higher downstream scores. For others, there was no meaningful relationship. This finding has been replicated and extended multiple times. The message is clear: intrinsic benchmarks like analogy accuracy are not reliable proxies for downstream task performance.

Why might a model with high analogy accuracy perform poorly on downstream tasks? One reason is task mismatch. The analogy dataset emphasizes encyclopedic and morphological relationships that appear frequently in Wikipedia-style text. A model trained on that distribution will learn those relationship types well, but might perform poorly on sentiment analysis (where emotional associations matter), medical NLP (where domain-specific terminology determines performance), or social media text classification (where informal language dominates). The analogy benchmark is not representative of the full space of language use.

Another reason is the nature of linear structure itself. High analogy accuracy indicates the model has learned linear subspaces for specific relationship types. But many downstream tasks require nonlinear reasoning, multi-word compositions, or document-level understanding that simple embedding lookups cannot provide. A model optimized for linear analogy accuracy is optimized for a very specific geometric property that may not transfer to tasks requiring richer representations.

The analogy datasets encode cultural and demographic biases that become embedded in the evaluation itself. The Google Analogy Dataset was constructed by manually selecting word pairs, and this selection reflects the perspectives and assumptions of its creators.

Family relationship analogies use binary gender categories: man:woman, brother:sister, king:queen. This reflects a particular cultural framing of gender as a binary categorical variable encoded in language, which may not generalize across languages, time periods, or communities. Currency pair analogies favor widely-known currencies, biasing toward wealthy and prominent nations. Many currencies of smaller countries are not represented.

The most pointed example is occupational analogies. If a dataset includes analogies like "man:doctor::woman:nurse," evaluating on that dataset rewards models that have learned occupational gender stereotypes. High accuracy on such questions indicates the model has absorbed stereotypical associations from its training data, not that it has learned linguistically correct or socially neutral representations. This confounds two very different kinds of knowledge: linguistic structure and cultural stereotypes.

This is not a reason to abandon analogy evaluation entirely. It is a reason to interpret analogy accuracy as evidence about what patterns the training data contained, not as evidence about neutral linguistic competence. A model that scores well on biased analogies has learned the biases in the corpus, which may be useful to know. A model that scores poorly on biased analogies might have undergone debiasing, or it might simply lack coverage. The accuracy number alone cannot distinguish these cases.

A model achieving 75% analogy accuracy has demonstrably learned linear subspaces corresponding to certain semantic and syntactic relationships. This is meaningful evidence. It indicates the model has captured some distributed structure from the training data, and that this structure is geometrically organized in a way that allows relationship-based reasoning. The fact that "king - man + woman" lands near "queen" tells you something real about the embedding geometry.

But it doesn't mean the embeddings represent language understanding in a general sense. It doesn't mean they will transfer to all tasks. It doesn't mean they are unbiased. And it doesn't mean a model that achieves 80% accuracy is better for your application than one that achieves 70%. Analogy evaluation is one lens among many, and it is most useful when combined with extrinsic evaluations on real tasks, similarity benchmarks on human judgment data, and qualitative analysis of the embeddings themselves.

The first plot illustrates a non-monotonic relationship between vocabulary size and analogy accuracy. With too few words, many analogy questions fall outside the vocabulary and are skipped, artificially reducing coverage while inflating the accuracy rate on attempted questions. With too many words, rare words enter the vocabulary as additional distractor candidates, making it harder for the correct answer to rank first. An intermediate vocabulary size around 50-100k words tends to maximize analogy accuracy for typical English training corpora.

The second plot reveals the weak correlation between analogy accuracy and downstream task performance. The scattered pattern means that knowing a model's analogy accuracy tells you little about how well it will perform on tasks like sentiment analysis, parsing, or information extraction. Models with identical analogy accuracy can differ dramatically in downstream performance depending on what other properties their embeddings capture.

Let's see the full analogy evaluation pipeline working end to end. We will create a realistic set of synthetic pre-trained embeddings with explicitly planted relational structure, evaluate both 3CosAdd and 3CosMul, and analyze where each method succeeds and fails.

The synthetic setup is important for a pedagogical reason: with real pre-trained embeddings, the embedding values are fixed and opaque. With synthetic embeddings where we control the relational structure, we can examine exactly what the algorithms find and why.

Both methods achieve high accuracy on this synthetic dataset because the relational offsets were explicitly planted and the noise level is controlled. Any errors occur when the noise in the planted offsets is large enough to push the target vector closer to a distractor word than to the correct answer. In real embeddings, this kind of noise is pervasive: the "gender direction" inferred from man/woman is not identical to the direction inferred from king/queen, because each pair accumulates its offset from a different subset of the training corpus.

The key insight for practical use is that this pipeline generalizes directly to real embeddings. Load any pre-trained embeddings (from Gensim, HuggingFace, or your own training), provide analogy triples in the same format, and evaluate both methods to understand which relationship types the embeddings capture. Comparing the category-level accuracy profile across different embedding configurations tells you far more than comparing a single aggregate number.

The analogy framework extends naturally to more complex reasoning patterns, though with diminishing reliability as the chains grow longer. If the embedding space encodes the capital-of relationship as a consistent offset $\vec{r}_{cap}$, then you can chain relationships: if France is to Paris as Germany is to Berlin, and Paris is to Eiffel Tower as Berlin is to Brandenburg Gate, then you might try to compose the two offsets to answer: "France is to Eiffel Tower as Germany is to ?". This is relationship composition: applying a sequence of relational offsets to follow multi-hop paths in the semantic graph.

In practice, composed analogies work less reliably than single-hop analogies because each step introduces noise. The first offset might point 5 degrees away from perfect, and the second offset adds another 5 degrees of error, compounding the directional drift. After two hops, the target vector may be far enough from the correct answer that a different word is the nearest neighbor.

Researchers have explored using analogy arithmetic for more structured reasoning tasks. Knowledge base completion, where the goal is to predict missing relationships in a structured knowledge graph, can be framed as analogy completion. Given (France, capital-of, Paris) and (Germany, capital-of, ?), the embedding space can sometimes predict Berlin. This approach, explored in TransE and related models, treats relationship types as translation vectors in the embedding space and explicitly trains the model to satisfy the parallelogram structure.

The success of TransE and similar approaches shows that the analogy insight generalizes beyond evaluation: if you design training objectives that explicitly enforce parallelogram structure for known relationships, you can achieve much stronger analogy performance on structured knowledge tasks. The Word2Vec and GloVe analogy results emerge as a byproduct of distributional training; TransE-style models achieve it as a primary objective.

Word analogy evaluation had enormous influence on the NLP field and on how practitioners think about embedding quality. It provided a simple, reproducible procedure: train embeddings, run the Google analogy task, report top-1 accuracy. This standardization accelerated research by creating a common benchmark where one model could be claimed to "beat" another with a single number. The ease of the comparison drove rapid model iteration.

The impact on model development was equally significant. Analogy performance became a concrete optimization target. Researchers tuned hyperparameters, compared training corpora, and modified architectures with analogy accuracy as a guide. Word2Vec, GloVe, and FastText all achieved progressively higher analogy scores through this iterative process. The benchmark became synonymous with embedding quality in the minds of many practitioners.

But the field has since developed a more qualified understanding. The limitations described in this chapter, particularly the weak correlation with downstream task performance, have shifted how serious practitioners evaluate embeddings. The current consensus in NLP research is that intrinsic evaluations like analogy accuracy are useful for rapid development feedback and sanity checking, but downstream task performance is the ultimate judge. A model that achieves 80% analogy accuracy but performs poorly on your specific task is not a good model for your application.

The analogy evaluation story also illustrates a broader methodological lesson about benchmark-driven research. When a benchmark becomes the primary optimization target, researchers find ways to improve performance on that benchmark, sometimes at the expense of other capabilities. The benchmark shapes what gets built. This pattern, sometimes called Goodhart's Law in the context of machine learning, is why the NLP field has progressively moved toward diverse evaluation suites, held-out test sets that are harder to overfit, and task-specific evaluation protocols.

Despite its limitations, word analogy evaluation remains a useful diagnostic tool for understanding the structure of embedding spaces. When you are training or selecting embeddings for a project, testing a handful of analogy questions quickly reveals whether basic semantic structure is present. If "king - man + woman" lands near "queen," the embeddings have learned gender as a consistent geometric direction. If "France + [capital direction]" lands near "Paris," they have learned geographic relationships. These are meaningful sanity checks that can catch obvious problems in training data, preprocessing, or hyperparameter choices.

The geometric insight underlying analogy evaluation also extends to more recent work on understanding language models. The idea that relationships can be encoded as directional offsets in a high-dimensional space reappears in interpretability research on transformer attention heads, in the design of controlled text generation methods, and in the analysis of how large language models represent factual knowledge. The parallelogram model may be a limited evaluation tool, but the underlying geometric intuition for how meaning can be organized in vector spaces remains one of the most productive ideas in modern NLP.

Word analogy evaluation reveals how well an embedding space captures semantic and syntactic relationships as linear vector offsets. The central ideas from this chapter are:

• The parallelogram model assumes semantic relationships encode as consistent directional offsets: $\vec{b} - \vec{a} \approx \vec{d} - \vec{c}$ for any two word pairs sharing the same relationship. This structure emerges naturally from distributional training because words in analogous roles appear in analogous contexts.

• Vector arithmetic predicts the answer word by computing $\vec{b} - \vec{a} + \vec{c}$ and finding the nearest vocabulary word. The algebraic simplicity of this operation is one of the most striking properties of well-trained word embeddings.

• 3CosAdd searches for the word maximizing cosine similarity to the target vector, equivalently written as $\cos(\vec{d}, \vec{b}) - \cos(\vec{d}, \vec{a}) + \cos(\vec{d}, \vec{c})$. The explicit form reveals the selection criterion: find a word that resembles $b$, resembles $c$, but does not resemble $a$.

• 3CosMul uses a multiplicative formulation $\frac{\cos(\vec{d}, \vec{b}) \cdot \cos(\vec{d}, \vec{c})}{\cos(\vec{d}, \vec{a}) + \epsilon}$ that avoids negative scores and is less sensitive when cosine similarities are near zero. It performs better on syntactic analogies.

• Cosine similarity is preferred over Euclidean distance because it is magnitude-invariant, measuring directional alignment rather than vector length differences.

• The Google Analogy Dataset, MSR Syntactic Dataset, and BATS provide standardized evaluation across semantic and syntactic relationship types, with BATS offering the finest diagnostic granularity.

• Analogy accuracy correlates weakly with downstream task performance, so analogy evaluation is best used as a fast development diagnostic rather than a primary quality measure. Optimizing for analogy accuracy does not reliably improve performance on classification, parsing, or generation tasks.

• Analogies fail on irregular morphology, polysemous words, and relationships that aren't linearly encodable. High accuracy indicates the embedding space has linear relational structure for the tested categories; it does not imply general linguistic competence.

• Analogy datasets encode cultural biases, so high accuracy on biased analogy pairs indicates the model has absorbed those biases from training data, not that it has learned neutral linguistic representations.

In the next chapter, we explore GloVe, which takes a fundamentally different approach to learning embeddings: instead of predicting context words one at a time, it directly factorizes the global co-occurrence matrix built from the entire corpus. GloVe's design will illuminate why both predictive and count-based methods can achieve similar analogy performance despite their very different training objectives, and what that convergence tells us about the information content of distributional statistics.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about word analogy and vector arithmetic in embedding spaces.