Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Embedding Evaluation › Analogy Evaluation

Michael BrenndoerferUpdated March 21, 20267 min read

Language AI HandbookAll chapters

Part 4 of 67Word Embeddings

Chapter 27Embedding Evaluation › Analogy Evaluation

Reading section 4 of 13 · Analogy Evaluation · 7 minSection 4 of 13 · 7 min

Section 4 of 13 · 7 min read

Embedding Evaluation › Analogy EvaluationOne section · Sections in this chapter4 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Beyond pairwise similarity, analogy tests assess whether embeddings encode structured semantic relationships rather than just proximity. The classic example is the gender analogy: the relationship between "king" and "queen" should be the same as between "man" and "woman," meaning the vector from "man" to "woman" should be approximately parallel to the vector from "king" to "queen." If embeddings have this property, vector arithmetic becomes a powerful tool for relational reasoning.

This property is remarkable because the model wasn't explicitly trained to learn relations. Word2Vec and GloVe learn from co-occurrence statistics. The fact that relational structure emerges from simple distributional learning is one of the key insights that made word embeddings such an exciting development.

The 3CosAdd MethodLink Copied

The standard analogy evaluation method is 3CosAdd. Given an analogy "a : b :: c : ?", you find the word $d$ that maximizes:

\text{3CosAdd}(a, b, c) = \arg\max_{d \in V \setminus \{a, b, c\}} \left[ \cos(\mathbf{v}_d, \mathbf{v}_b) - \cos(\mathbf{v}_d, \mathbf{v}_a) + \cos(\mathbf{v}_d, \mathbf{v}_c) \right]

where:

$V$ is the full vocabulary
$\mathbf{v}_a$ , $\mathbf{v}_b$ , $\mathbf{v}_c$ , $\mathbf{v}_d$ are the embeddings of words $a$ , $b$ , $c$ , and candidate $d$
$\cos(\mathbf{u}, \mathbf{v}) = \frac{\mathbf{u} \cdot \mathbf{v}}{|\mathbf{u}||\mathbf{v}|}$ is the cosine similarity
The $\arg\max$ ranges over all vocabulary words except the three query words

This scoring function is equivalent to finding $d$ whose embedding is closest to $\mathbf{v}_b - \mathbf{v}_a + \mathbf{v}_c$ . The subtraction $\mathbf{v}_b - \mathbf{v}_a$ extracts the relationship vector: for "man : king", this is roughly a "royalty" direction. Adding $\mathbf{v}_c$ (e.g., "woman") to this royalty direction should land you near "queen." The three cosine terms capture this arithmetic while handling unit-normalized vectors more cleanly than raw vector arithmetic.

Importantly, the answer candidates exclude $a$ , $b$ , and $c$ themselves, since the trivial solution would often be one of those words. If you didn't exclude them, "king" would frequently be the top answer for the analogy "man : king :: woman : ?" simply because "king" scores highly in cosine similarity to the target vector.

Why 3CosAdd Works (and Sometimes Fails)Link Copied

The geometric intuition behind 3CosAdd is clean, but the method has known failure modes. The most important is frequency bias: very frequent words (function words, generic nouns) tend to appear as top neighbors for many queries simply because they have dense co-occurrence patterns with many other words. These words can become "hubs" that monopolize top- $k$ neighbor lists across the embedding space. When a hub word happens to be the correct analogy answer, the method looks brilliant; when it's not, the hub crowds out the right answer.

A second failure mode is the linearity assumption itself. Vector arithmetic assumes that the relationship between "man" and "woman" is the same linear transformation as the relationship between "king" and "queen." This works well for some relations (gender, country-capital, singular-plural) but breaks down for more complex semantic relationships. The analogy "Paris : France :: Berlin : Germany" works beautifully because both involve the "capital of" relation. But "good : better :: bad : worse" works because comparative form is a linear morphological operation. Relations that don't decompose linearly in the embedding space will yield poor analogy accuracy even when the embeddings are otherwise excellent.

The 3CosMul method (Levy and Goldberg, 2014) addresses some of these issues by using a multiplicative combination rather than additive:

\text{3CosMul}(a, b, c) = \arg\max_{d} \frac{\cos(\mathbf{v}_d, \mathbf{v}_b) \cdot \cos(\mathbf{v}_d, \mathbf{v}_c)}{\cos(\mathbf{v}_d, \mathbf{v}_a) + \epsilon}

where $\epsilon$ is a small constant to prevent division by zero. 3CosMul is less sensitive to the hub problem and often outperforms 3CosAdd on semantic analogies, though 3CosAdd remains the more commonly reported metric for historical reasons.

The Google Analogy DatasetLink Copied

The most widely used analogy benchmark is the Google Analogy Dataset (Mikolov et al., 2013), containing 19,544 analogy questions in two categories:

Semantic analogies (8,869 questions): capital-country ("Paris : France :: Berlin : Germany"), capital-world, currency, city-in-state, family relationships
Syntactic analogies (10,675 questions): comparative adjectives ("good : better :: bad : worse"), superlatives, past tense, plural forms, nationalities

The syntactic/semantic split is analytically useful. Embeddings that score much higher on syntactic analogies than semantic ones likely capture morphological and grammatical patterns well but struggle with world knowledge. FastText, which builds subword representations, typically excels at syntactic analogies because it can represent morphological relationships even for rare words. GloVe, trained on document-level co-occurrence, often scores higher on semantic analogies because document co-occurrence patterns capture topical associations.

Let's implement the evaluation:

In[7]:

Code

def find_analogy(embeddings, word_a, word_b, word_c, top_n=5):
    """
    Solve analogy: word_a is to word_b as word_c is to ?
    Uses 3CosAdd method: find word closest to v_b - v_a + v_c
    """
    vocab = list(embeddings.keys())

    # Check all words are in vocabulary
    for word in [word_a, word_b, word_c]:
        if word not in embeddings:
            return None, []

    v_a = embeddings[word_a]
    v_b = embeddings[word_b]
    v_c = embeddings[word_c]

    # Target vector: v_b - v_a + v_c
    target = v_b - v_a + v_c
    target = target / np.linalg.norm(target)

    # Score all vocabulary words
    exclude = {word_a, word_b, word_c}
    scores = []

    for word in vocab:
        if word in exclude:
            continue
        sim = cosine_similarity(target, embeddings[word])
        scores.append((word, sim))

    scores.sort(key=lambda x: x[1], reverse=True)
    top_predictions = scores[:top_n]

    return top_predictions[0][0], top_predictions


# Example analogies using our structured embeddings
analogy_examples = [
    ("cat", "kitten", "dog"),  # Expected: puppy
    ("car", "automobile", "truck"),  # Expected: similar vehicle synonym
    ("computer", "laptop", "phone"),  # Expected: tech-related
]

Out[8]:

Console

Analogy Examples (3CosAdd method)
============================================================
Format: A : B :: C : ? (expected answer)

  cat : kitten :: dog : ?
  Top predictions: ['puppy (0.866)', 'horse (0.699)', 'computer (0.112)']

  car : automobile :: truck : ?
  Top predictions: ['bus (0.885)', 'bike (0.862)', 'phone (0.140)']

  computer : laptop :: phone : ?
  Top predictions: ['program (0.974)', 'software (0.969)', 'car (0.176)']

In[9]:

Code

def evaluate_analogies(embeddings, analogies):
    """
    Evaluate embedding quality on a set of analogy questions.
    Each analogy is (word_a, word_b, word_c, correct_answer).
    """
    correct = 0
    total = 0
    skipped = 0

    results = []

    for word_a, word_b, word_c, expected in analogies:
        # Skip if any word missing
        if any(w not in embeddings for w in [word_a, word_b, word_c, expected]):
            skipped += 1
            continue

        total += 1
        prediction, _ = find_analogy(embeddings, word_a, word_b, word_c)
        is_correct = prediction == expected
        if is_correct:
            correct += 1
        results.append(
            {
                "analogy": f"{word_a}:{word_b}::{word_c}:{expected}",
                "predicted": prediction,
                "correct": is_correct,
            }
        )

    accuracy = correct / total if total > 0 else 0
    return accuracy, results, skipped


# A small test analogy set (word_a, word_b, word_c, expected_answer)
test_analogies = [
    ("cat", "kitten", "dog", "puppy"),
    ("car", "truck", "cat", "horse"),
    ("computer", "software", "car", "bus"),
]

accuracy, analogy_results, n_skipped = evaluate_analogies(
    embeddings, test_analogies
)

Out[10]:

Console

Analogy Evaluation Results
=======================================================
Total questions: 3
Skipped (OOV):   0
Accuracy:        33.3%

  [CORRECT] cat:kitten::dog:puppy
         Predicted: puppy

  [WRONG] car:truck::cat:horse
         Predicted: kitten

  [WRONG] computer:software::car:bus
         Predicted: automobile

State-of-the-art Word2Vec models achieve around 60-70% accuracy on the Google Analogy Dataset, with generally higher accuracy on syntactic analogies than semantic ones. GloVe typically performs similarly. FastText improves on syntactic analogies due to its subword structure, which makes it better at morphological relations like singular-plural and comparative adjectives.

What Analogy Accuracy Tells YouLink Copied

Analogy accuracy is an attractive metric because it seems to directly test relational reasoning. But there are important caveats. The metric is highly sensitive to vocabulary coverage: if any of the four words in an analogy is missing from the embedding vocabulary, the entire question is skipped. Models with larger vocabularies cover more questions and compute accuracy on a larger, more representative sample. Always report both the accuracy and the coverage rate.

The accuracy metric also assumes exactly one correct answer. For "Paris : France :: Berlin : ?", "Germany" is the expected answer, but "German" or "Germanic" might be reasonable alternatives. Real analogy datasets often have quirks in their expected answers that penalize valid reasoning. And because 3CosAdd searches the entire vocabulary, models with very large vocabularies face a harder search problem than models with small vocabularies, which can make accuracy numbers across models hard to compare directly.