Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Word Analogy › The 3CosAdd Method

Michael BrenndoerferApril 5, 20257 min read

Language AI HandbookAll chapters

Part 4 of 67Word Embeddings

Chapter 24Word Analogy › The 3CosAdd Method

Reading section 3 of 13 · The 3CosAdd Method · 7 minSection 3 of 13 · 7 min

Section 3 of 13 · 7 min read

Word Analogy › The 3CosAdd MethodOne section · Sections in this chapter3 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

We have established the intuition: semantic relationships create parallel vectors, and we can complete analogies by adding relationship offsets. But how do we find the answer word across tens of thousands of vocabulary items? The target vector we compute is a point in high-dimensional space. We need to find the vocabulary word whose embedding lies closest to that point, and we need a distance measure that is appropriate for the geometry of embedding spaces.

The standard approach searches for the vocabulary word $d$ that maximizes cosine similarity to the target vector $\vec{b} - \vec{a} + \vec{c}$ :

d^* = \arg\max_{d \in V \setminus \{a,b,c\}} \cos\!\left(\vec{d},\ \vec{b} - \vec{a} + \vec{c}\right)

where:

$V$ : the full vocabulary
$\{a, b, c\}$ : excluded because we want a fourth, distinct word as the answer
$\cos(\vec{x}, \vec{y})$ : the cosine similarity between vectors $\vec{x}$ and $\vec{y}$

This formulation is called 3CosAdd because it uses three vectors summed additively before computing cosine similarity. The name also reflects how it is equivalently written as a sum of three individual cosine similarities, which we'll derive below.

Why Cosine Similarity, Not Euclidean Distance?Link Copied

You might wonder why we use cosine similarity rather than simply finding the word with minimum Euclidean distance to the target vector. The problem is that word embeddings have different magnitudes. Frequent words like "the," "a," and "is" appear in enormous numbers of training contexts and tend to receive large embedding norms because their gradient updates accumulate over many training examples. Rare words like "czar" or "serration" appear in few contexts and have smaller norms. Euclidean distance conflates directional alignment with magnitude matching: it would favor words whose norms happen to be similar to the target's magnitude, even if they point in a completely different direction.

Cosine similarity solves this by focusing purely on the angle between vectors, ignoring their lengths entirely. It measures the cosine of the angle between two vectors:

\cos(\vec{x}, \vec{y}) = \frac{\vec{x} \cdot \vec{y}}{\|\vec{x}\| \cdot \|\vec{y}\|}

where:

$\vec{x} \cdot \vec{y}$ : the dot product of the vectors, computed as the sum of element-wise products $\sum_i x_i y_i$
$\|\vec{x}\|$ , $\|\vec{y}\|$ : the Euclidean norms (lengths) of the vectors, computed as $\sqrt{\sum_i x_i^2}$ and $\sqrt{\sum_i y_i^2}$

The denominator normalizes by both vector lengths, making the result range from -1 to 1 regardless of magnitude. Two vectors pointing in exactly the same direction have cosine similarity 1. Perpendicular vectors have cosine similarity 0. Opposite-pointing vectors yield -1. For analogy completion, we care about which word's embedding points the same direction as our target, not which word has a similar magnitude.

There is one subtlety. When we compute $\vec{b} - \vec{a} + \vec{c}$ , the resulting target vector is not normalized. It is a raw algebraic combination of three embedding vectors. Cosine similarity then compares this unnormalized target against each candidate in the vocabulary. The comparison is still direction-based, which is what we want, because cosine similarity normalizes both arguments before measuring the angle.

In[7]:

Code

def cosine_similarity(v1, v2):
    """Compute cosine similarity between two vectors."""
    dot = np.dot(v1, v2)
    norm = np.linalg.norm(v1) * np.linalg.norm(v2)
    return dot / norm if norm > 0 else 0.0


def three_cos_add(a, b, c, embeddings, exclude=None):
    """
    Solve analogy a:b :: c:? using 3CosAdd.
    Returns ranked list of (word, similarity) pairs.
    """
    if exclude is None:
        exclude = {a, b, c}
    target = embeddings[b] - embeddings[a] + embeddings[c]
    scores = {}
    for word, vec in embeddings.items():
        if word not in exclude:
            scores[word] = cosine_similarity(target, vec)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)


# Solve: man:woman :: king:?
results = three_cos_add("man", "woman", "king", embeddings_2d)

Out[8]:

Console

Analogy: man is to woman as king is to ?
Ranking by 3CosAdd (cosine similarity to target):

  1. queen        cos=1.0000  [====================]
  2. princess     cos=0.9991  [===================]
  3. aunt         cos=0.9926  [===================]
  4. prince       cos=0.8382  [================]
  5. uncle        cos=0.1843  [===]

Queen ranks first because the synthetic embeddings were constructed with perfect parallelogram structure. In real embeddings trained on text, the correct answer typically ranks first or second for well-studied relationship types, with errors most common on ambiguous, rare, or morphologically irregular words.

The 3CosAdd EquivalenceLink Copied

3CosAdd can be rewritten in an equivalent form that makes its logic more transparent. Rather than computing cosine similarity to a single composite target vector, 3CosAdd turns out to maximize a combination of three individual cosine similarities. This equivalence reveals exactly what the algorithm is selecting for.

The derivation begins with the observation that cosine similarity is linear in its first argument when the second argument is fixed. Expanding $\cos(\vec{d},\ \vec{b} - \vec{a} + \vec{c})$ and exploiting the bilinearity of the dot product yields the equivalent form:

d^* = \arg\max_{d} \left[\cos(\vec{d}, \vec{b}) - \cos(\vec{d}, \vec{a}) + \cos(\vec{d}, \vec{c})\right]

where:

$\cos(\vec{d}, \vec{b})$ : reward words similar to $b$ , the "answer template" providing the target semantic role
$-\cos(\vec{d}, \vec{a})$ : penalize words similar to $a$ , subtracting the source concept
$\cos(\vec{d}, \vec{c})$ : reward words similar to $c$ , the query word we are completing

This reformulation reveals the algorithm's selection criterion more clearly than the vector addition form. You want a word that resembles $b$ (in the same role as the target), resembles $c$ (related to the query), but does not resemble $a$ (not the same as the source). For "man is to woman as king is to ?", you want a word that is like "woman" (feminine), like "king" (royal), but not like "man" (masculine). "Queen" satisfies all three conditions more strongly than any other word in the vocabulary.

The subtraction term determines the result. Without it, you would just find the word most similar to both "woman" and "king", which might be "ruler" or "monarch" or some other word that is royal and somewhat gender-associated. The subtraction penalizes "man" and similar male terms, forcing the algorithm to find a word that has the femininity of "woman" without having the maleness of "man."

Note that this equivalence is approximate. It holds exactly only when all vectors are unit-normalized, because the derivation assumes the cosine similarity of the sum equals the sum of cosine similarities. In practice, Word2Vec and GloVe embeddings are not perfectly normalized, so the two forms can give slightly different rankings. The explicit three-cosine form is sometimes preferred in practice because it can be more numerically stable.

In[9]:

Code

def three_cos_add_explicit(a, b, c, embeddings, exclude=None):
    """
    3CosAdd using the explicit three-cosine formulation.
    Equivalent to the vector addition approach.
    """
    if exclude is None:
        exclude = {a, b, c}
    scores = {}
    for word, vec in embeddings.items():
        if word not in exclude:
            score = (
                cosine_similarity(vec, embeddings[b])
                - cosine_similarity(vec, embeddings[a])
                + cosine_similarity(vec, embeddings[c])
            )
            scores[word] = score
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)


results_explicit = three_cos_add_explicit("man", "woman", "king", embeddings_2d)

Out[10]:

Console

3CosAdd explicit formulation: man is to woman as king is to ?

  1. princess     score=2.2585
  2. queen        score=2.2464
  3. aunt         score=2.1900
  4. prince       score=1.7045
  5. uncle        score=0.0923

Both formulations yield identical rankings on these normalized 2D embeddings, confirming they are mathematically equivalent in this case. When working with high-dimensional, unnormalized embeddings, the two forms can differ slightly. In practice, you can use either; the explicit form is often clearer when implementing from scratch and the vector addition form is faster for large vocabulary searches.