Word Analogy: Vector Arithmetic for Semantic Relationships

Something remarkable happens when you train Skip-gram or CBOW on billions of words: the resulting embeddings exhibit structured geometric relationships that mirror semantic relationships. The famous example: $\vec{king} - \vec{man} + \vec{woman} \approx \vec{queen}$. This isn't a quirk. The vector from "man" to "woman" captures a gender relationship, and adding it to "king" lands near "queen." The embedding space has learned that kingship relates to queenship the same way manhood relates to womanhood.

This phenomenon, known as word analogy, became a central way to evaluate and understand word embeddings. If an embedding space correctly solves "man is to woman as king is to ?", it suggests the model has captured meaningful semantic structure. But analogies also reveal limitations: they work well for certain relationship types and fail for others, and the popular evaluation metrics can be misleading.

This chapter explores the parallelogram model underlying word analogies, develops the mathematical methods for solving them (3CosAdd and 3CosMul), examines standard evaluation datasets, and discusses what analogies actually tell us about embedding quality.

Word analogies rely on a geometric intuition: if two word pairs share the same relationship, the vectors connecting them should be parallel. Consider the relationship "capital of":

• Paris is the capital of France
• Tokyo is the capital of Japan
• London is the capital of England

If embeddings capture this relationship consistently, then $\vec{Paris} - \vec{France}$, $\vec{Tokyo} - \vec{Japan}$, and $\vec{London} - \vec{England}$ should all point in roughly the same direction. This directional consistency allows us to complete analogies.

The parallelogram structure is idealized. Real embeddings have noise, and the offsets aren't perfectly parallel. But when embeddings are trained well, the relationship vectors are similar enough that analogy completion works.

The classic analogy task asks: "a is to b as c is to ?" We need to find a word $d$ that completes the analogy. The key insight is that if the relationship between $a$ and $b$ is the same as the relationship between $c$ and $d$, then the vector offsets should be equal. Rearranging this gives us the target vector:

where:

• $\vec{a}$, $\vec{b}$, $\vec{c}$, $\vec{d}$: embedding vectors for words $a$, $b$, $c$, and $d$
• $\vec{b} - \vec{a}$: the relationship vector capturing how $b$ differs from $a$
• $\vec{c}$: the starting point for our new word pair

The intuition behind this formula:

1. Compute the relationship vector: $\vec{b} - \vec{a}$ captures how $b$ differs from $a$
2. Apply this offset to $c$: adding $\vec{b} - \vec{a}$ to $\vec{c}$ should land near $d$
3. Find the nearest word: search the vocabulary for the word closest to the computed vector

The distance of zero confirms that our synthetic embeddings exhibit perfect parallelogram structure: the computed target vector exactly matches the embedding for "queen." In real embeddings, there's noise, so we need to find the nearest vocabulary word.

We've established the intuition: semantic relationships create parallel vectors, and we can complete analogies by adding relationship offsets. But how do we actually find the answer word? In our 2D visualization, we could simply look at where the target vector lands. In 50 or 300 dimensions, we need a principled search procedure.

The challenge is that the target vector $\vec{b} - \vec{a} + \vec{c}$ almost never exactly equals any word's embedding. Real embeddings have noise, relationships aren't perfectly parallel, and we're working in high-dimensional spaces where geometric intuitions can mislead. What we need is a way to find the closest word to our computed target.

You might think: just find the word with minimum Euclidean distance to the target vector. But consider what happens when embeddings have different magnitudes. A frequent word like "the" might have a large embedding norm, while a rare word like "czar" has a small norm. Euclidean distance would favor words whose norms happen to match the target, regardless of directional similarity.

Cosine similarity solves this by focusing purely on direction. It measures the cosine of the angle between two vectors:

where:

• $\vec{x}$, $\vec{y}$: the two vectors being compared
• $\vec{x} \cdot \vec{y}$: the dot product of the vectors (sum of element-wise products)
• $||\vec{x}||$, $||\vec{y}||$: the Euclidean norms (lengths) of the vectors

The denominator normalizes the dot product by the vector lengths, making the result independent of magnitude. Two vectors pointing in the same direction have cosine similarity 1, regardless of their lengths. Perpendicular vectors have similarity 0. Opposite directions yield -1. For analogy completion, we care about direction (does this word's embedding point the same way as our target?), not magnitude.

Given the analogy "a is to b as c is to ?", the 3CosAdd method searches the entire vocabulary to find the word whose embedding is most similar (by cosine) to our target vector. The method gets its name from combining three terms additively: we add $\vec{b}$ and $\vec{c}$ while subtracting $\vec{a}$:

where:

• $d^*$: the optimal answer word (the word that maximizes the score)
• $\arg\max$: returns the word $d$ that achieves the maximum value
• $V$: vocabulary (the set of all words with embeddings)
• $V \setminus \{a, b, c\}$: vocabulary excluding the query words
• $a$, $b$, $c$: the query words in the analogy "$a$ is to $b$ as $c$ is to ?"
• $\vec{d}$: embedding vector for candidate word $d$
• $\vec{b} - \vec{a} + \vec{c}$: the target vector computed by vector arithmetic
• $\cos(\cdot, \cdot)$: cosine similarity function

Notice the exclusion of query words ($V \setminus \{a, b, c\}$): without this, "woman" might rank highest for "man:woman::king:?" because it's already part of the target vector computation. We want to discover the answer, not echo back the inputs.

The name "3CosAdd" comes from viewing the formula as three additive cosine operations. Conceptually, we want a word that is:

1. Similar to $b$ (shares properties with the "answer" exemplar)
2. Similar to $c$ (shares properties with our new starting point)
3. Dissimilar to $a$ (doesn't share properties we're subtracting away)

The vector arithmetic $\vec{b} - \vec{a} + \vec{c}$ combines these requirements into a single target.

Let's build the algorithm from scratch. First, we need a function to compute cosine similarity between any two vectors:

The implementation handles the edge case of zero-magnitude vectors (which shouldn't occur in practice but guards against numerical issues). Now the main algorithm:

The algorithm is straightforward: compute the target, score every word in the vocabulary, and return a ranked list. The computational cost is $O(|V| \cdot d)$ where $|V|$ is vocabulary size and $d$ is embedding dimension, since we must check every word. For large vocabularies, this can be accelerated using approximate nearest neighbor search, but the exact search remains common for evaluation.

The 3CosAdd method correctly identifies "queen" as the best answer with the highest similarity score. Notice that other female terms (woman, princess, aunt) also score relatively high because they share the "female" component with the target vector, even though they lack the "royalty" component that specifically makes "queen" the best match.

This reveals something important: 3CosAdd doesn't require a perfect match. It finds the best available word, which works well when the vocabulary contains the expected answer. When the vocabulary lacks the expected word (perhaps the analogy expects "empress" but only "queen" is available), the method gracefully returns the closest alternative.

Let's load pre-trained GloVe embeddings and test analogies on real data:

The results demonstrate that 3CosAdd successfully captures the semantic relationships we encoded in our synthetic embeddings. Analogies where the expected answer appears at rank 1 are correct, while higher ranks indicate that other words scored higher than the expected answer. The overall accuracy indicates how consistently the embedding space encodes the relationship patterns we designed.

3CosAdd works well, but it has a subtle weakness. Consider what happens when one of the similarity terms is unusually large. If a candidate word happens to be extremely similar to $b$, this high similarity can dominate the vector arithmetic, potentially overriding the contributions from $a$ and $c$. The additive combination means one term can "swamp" the others.

Levy and Goldberg (2014) proposed an elegant alternative: instead of adding the effects of each query word into a single target vector, treat them separately and combine with multiplication. This is 3CosMul, the multiplicative analogy method.

Think about what we want in an answer word $d$:

• It should be similar to $b$ (the "answer" in our template pair)
• It should be similar to $c$ (our new starting point)
• It should be dissimilar to $a$ (the word we're "subtracting")

With multiplication, all three conditions must be satisfied simultaneously. A word that's perfect for two conditions but terrible for the third gets a low score because multiplication drives the result toward zero. With addition, a single excellent match could compensate for poor matches elsewhere.

Consider a concrete example: for "man:woman::king:?", suppose there's a word "she" that's extremely similar to "woman" (similarity 0.95) but not particularly related to royalty. Under 3CosAdd, this high similarity to "woman" might boost "she" above "queen." Under 3CosMul, "she" would need to also score well on similarity to "king" and dissimilarity to "man", requirements it's unlikely to satisfy.

The 3CosMul method treats the three similarity requirements separately and combines them multiplicatively. Instead of computing a single target vector and measuring distance to it, 3CosMul computes three separate similarity scores for each candidate word:

where:

• $d^*$: the optimal answer word (the word that maximizes the score)
• $\arg\max$: returns the word $d$ that achieves the maximum value
• $V \setminus \{a, b, c\}$: vocabulary excluding the query words
• $\text{sim}(\vec{x}, \vec{y}) = \frac{\cos(\vec{x}, \vec{y}) + 1}{2}$: shifted cosine similarity mapped to $[0, 1]$
• $\epsilon$: small constant (typically 0.001) to prevent division by zero
• Numerator: $\text{sim}(\vec{d}, \vec{b}) \cdot \text{sim}(\vec{d}, \vec{c})$ rewards words similar to both $b$ and $c$
• Denominator: $\text{sim}(\vec{d}, \vec{a}) + \epsilon$ penalizes words similar to $a$

The structure mirrors our intuition: multiply the "want" terms (similarity to $b$ and $c$), divide by the "don't want" term (similarity to $a$). The $\epsilon$ prevents division by zero when a candidate is perfectly orthogonal to $a$.

There's a technical subtlety: cosine similarity ranges from -1 to +1. Multiplying two negative numbers gives a positive result, which would incorrectly reward candidates that are dissimilar to both $b$ and $c$.

The solution is to shift similarities to a positive range before multiplying. Instead of using raw cosine similarities in $[-1, 1]$, we apply a linear transformation to map them to $[0, 1]$:

where:

• $\cos(\vec{x}, \vec{y})$: the original cosine similarity in range $[-1, 1]$
• $+1$: shifts the range from $[-1, 1]$ to $[0, 2]$
• $/2$: scales the range from $[0, 2]$ to $[0, 1]$

This transformation preserves the ordering of similarities while ensuring all values are positive. Now a cosine similarity of $-1$ becomes $0$, a similarity of $0$ becomes $0.5$, and a similarity of $1$ becomes $1$. Multiplication works correctly: high positive similarities contribute large factors, while negative similarities (now mapped to small positive values) appropriately dampen the score.

The implementation follows our formula, with care taken to shift similarities to the positive range:

Let's visualize how 3CosMul scores each candidate word by examining the component similarities:

Now let's compare the two methods head-to-head on our test analogies:

On our synthetic embeddings with well-structured relationships, both methods typically produce similar results. The real differences between 3CosAdd and 3CosMul emerge in noisier, real-world embeddings where one query word's similarity might otherwise dominate the score.

On our synthetic embeddings with clean structure, both methods typically agree. But with real embeddings trained on noisy text, differences emerge:

1. Dominating similarities: When one query word is unusually similar to many candidates, 3CosAdd can be swayed by that single term. 3CosMul requires balanced similarity across all query words, making it more robust to outliers.

2. Syntactic relationships: Levy and Goldberg found that 3CosMul outperforms 3CosAdd specifically on syntactic analogies (verb tenses, plurals, comparatives). These relationships may have less consistent offsets than semantic relationships, making the multiplicative balancing more valuable.

3. Rare words: When one of the query words is rare and has a noisy embedding, its contribution to 3CosAdd's vector arithmetic might be unreliable. 3CosMul's separate treatment of each word can isolate this noise.

In practice, 3CosMul provides a small but consistent improvement on standard benchmarks. The Google analogy dataset shows roughly 2-3% better accuracy for 3CosMul, with gains concentrated in syntactic categories.

The NLP community has developed standard datasets for evaluating word embedding quality through analogies. These datasets contain thousands of analogy questions across various relationship categories.

The most famous benchmark, released with the original Word2Vec paper, contains 19,544 analogy questions in two categories:

Semantic analogies (8,869 questions):

• Capital-world: Athens is to Greece as Baghdad is to Iraq
• Capital-common: Beijing is to China as Berlin is to Germany
• Currency: Algeria is to dinar as Angola is to kwanza
• Family: boy is to girl as brother is to sister
• City-in-state: Chicago is to Illinois as Houston is to Texas

Syntactic analogies (10,675 questions):

• Adjective-to-adverb: apparent is to apparently as rapid is to rapidly
• Opposite: aware is to unaware as certain is to uncertain
• Comparative: bad is to worse as big is to bigger
• Superlative: bad is to worst as big is to biggest
• Present-participle: code is to coding as dance is to dancing
• Nationality-adjective: Albania is to Albanian as Argentina is to Argentinean
• Past-tense: dancing is to danced as decreasing is to decreased
• Plural: banana is to bananas as bird is to birds
• Plural-verbs: decrease is to decreases as describe is to describes

Performance varies across categories, reflecting how consistently each relationship type is encoded in the embedding space. Categories with higher accuracy indicate relationships that the embeddings capture more reliably. In real embeddings, semantic categories (like family relationships) often outperform syntactic ones (like verb tenses), though this varies by embedding method and training corpus.

Microsoft Research released another widely used dataset focusing on syntactic relationships:

• Adjectives: base, comparative, superlative forms
• Nouns: singular, plural forms
• Verbs: tense variations

BATS addresses limitations of earlier datasets by including:

• More diverse relationships (40 categories)
• Multiple valid answers per analogy
• Better balance between frequency levels

The standard metric for analogy evaluation is accuracy: the percentage of analogies where the top-ranked word (after excluding query words) matches the expected answer. This is a simple and intuitive metric that captures how often the model gets the analogy exactly right:

where:

• $\text{Accuracy}$: fraction of analogies answered correctly (ranges from $0$ to $1$, often expressed as a percentage)
• $N_{\text{correct}}$: number of analogies where the top-ranked prediction matches the expected answer
• $N_{\text{total}}$: total number of analogy questions evaluated

An accuracy of $0.75$ (or 75%) means the model correctly answered three out of every four analogies. However, this metric has important limitations.

Accuracy only considers the top-ranked answer. If the correct word is ranked second, it counts as wrong. This is particularly harsh for:

• Synonyms: "quick" and "fast" might both be valid
• Near-synonyms in context: "happy" and "joyful"
• Equally valid alternatives: both "woman" and "lady" might complete an analogy

The relaxed metrics show that many "incorrect" answers under strict Top-1 evaluation are actually ranked highly. Top-3 and Top-5 accuracy are often substantially higher than Top-1, indicating that the model often places the correct answer near the top even when it's not first. The MRR provides a single number that balances these considerations: higher MRR indicates correct answers consistently appear near the top of rankings.

MRR provides a more nuanced view by considering not just whether the correct answer is ranked first, but how high it appears in the ranked list. Instead of treating all non-first-place answers as failures, MRR gives partial credit based on rank position:

where:

• $\text{MRR}$: Mean Reciprocal Rank (ranges from $0$ to $1$)
• $N$: total number of analogy questions
• $\text{rank}_i$: position of the correct answer in the ranked list for the $i$-th analogy (1 if first, 2 if second, etc.)
• $\frac{1}{\text{rank}_i}$: reciprocal rank for analogy $i$ ($1$ if correct answer is first, $0.5$ if second, $0.33$ if third, etc.)
• $\sum_{i=1}^{N}$: sum over all $N$ analogies

The reciprocal rank gives high scores to correct answers near the top of the list and diminishing scores as rank increases. An MRR of $1.0$ means all answers were ranked first. An MRR of $0.5$ would occur if answers were typically ranked second.

Analogy performance tells us something about embedding quality, but the relationship is nuanced. Here's what good analogy performance does and doesn't imply.

Consistent relationship encoding: If "king:queen::man:woman" works, the model has learned that gender is encoded consistently across different word pairs. The relationship vector is approximately invariant.

Geometric structure: High analogy accuracy indicates the embedding space has meaningful geometric properties. Similar relationships create parallel vectors.

Distributional pattern learning: Since analogies emerge from distributional training (Skip-gram, GloVe), good performance confirms the model captured co-occurrence patterns that reflect semantic relationships.

General NLP performance: Analogy accuracy doesn't strongly predict performance on downstream tasks like sentiment analysis, named entity recognition, or question answering. Models can excel at analogies but underperform on practical applications, and vice versa.

Handling of rare words: Analogy datasets focus on common words. Performance on frequent words doesn't guarantee quality representations for the long tail of rare vocabulary.

Contextual understanding: Static embeddings give one vector per word. "Bank" in "river bank" and "bank account" gets the same embedding, yet analogy tests can't detect this limitation.

Compositionality: Analogies test individual word relationships, not how well embeddings combine into phrase or sentence representations.

To understand why some analogies work better than others, we can examine the relationship vectors directly. If a relationship is encoded consistently, all instances should produce similar offset vectors.

Relationship consistency measures how parallel the offset vectors are across different word pairs. Values closer to 1.0 indicate that all pairs in that relationship category share nearly identical offset directions, which strongly predicts good analogy performance. Lower consistency suggests the relationship is encoded differently for different word pairs, making analogies less reliable.

Despite their popularity, analogy tests have significant limitations as embedding quality metrics.

High-dimensional spaces suffer from "hubness": some words become nearest neighbors of many other words. These hubs can dominate analogy results, appearing as incorrect answers for many different queries.

Words that appear frequently as nearest neighbors ("hubs") can distort analogy results. If a hub word happens to score highly for many different analogies, it may incorrectly outrank the true answer. The disparity between central and peripheral words reveals the hubness structure of the embedding space, a potential source of systematic errors in analogy evaluation.

Common words have more training examples, leading to better-tuned embeddings. Analogy datasets often focus on frequent words, potentially overstating embedding quality for typical vocabulary.

Analogy datasets may contain patterns that don't generalize:

• Specific naming conventions (e.g., country capitals all having similar suffixes)
• Cultural biases (Western-centric knowledge)
• Temporal changes (city names, currencies that have changed)

Researchers have proposed several extensions to address limitations of the basic analogy framework.

Instead of requiring exact vector arithmetic to find an unknown word, relational similarity directly measures how well a model captures that two word pairs share the same relationship. Given two complete word pairs, it asks: "Do these pairs exhibit the same relationship?"

where:

• $\text{RelSim}(a:b, c:d)$: relational similarity between word pairs $(a, b)$ and $(c, d)$ (ranges from $-1$ to $1$)
• $\vec{b} - \vec{a}$: relationship vector from word $a$ to word $b$
• $\vec{d} - \vec{c}$: relationship vector from word $c$ to word $d$
• $\cos(\cdot, \cdot)$: cosine similarity function

A high relational similarity (close to $1$) indicates that both pairs exhibit the same relationship: the vectors $\vec{b} - \vec{a}$ and $\vec{d} - \vec{c}$ point in the same direction. This directly measures whether the relationship vectors are parallel, without requiring finding the correct $d$.