Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Word Analogy › The 3CosMul Method

Michael BrenndoerferApril 5, 20255 min read

Language AI HandbookAll chapters

Part 4 of 67Word Embeddings

Chapter 24Word Analogy › The 3CosMul Method

Reading section 4 of 13 · The 3CosMul Method · 5 minSection 4 of 13 · 5 min

Section 4 of 13 · 5 min read

Word Analogy › The 3CosMul MethodOne section · Sections in this chapter4 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

While 3CosAdd is the standard approach, it has a known vulnerability related to the subtraction operation. The term $-\cos(\vec{d}, \vec{a})$ penalizes words that are similar to $a$ , but when cosine similarity values are small and near zero, the arithmetic can amplify noise in unpredictable ways. A word that is slightly negatively correlated with $a$ for unrelated reasons will receive a spurious bonus. Conversely, a word that is the correct answer but happens to have mild positive similarity to $a$ will be penalized unfairly.

Levy, Goldberg, and Dagan (2014) proposed 3CosMul as an alternative that multiplies similarities rather than adding and subtracting them. The intuition is that multiplication is less sensitive to near-zero values: a word with low similarity to anything will score low overall, rather than getting an arbitrary bonus from a negative subtraction.

d^* = \arg\max_{d} \frac{\cos(\vec{d}, \vec{b}) \cdot \cos(\vec{d}, \vec{c})}{\cos(\vec{d}, \vec{a}) + \epsilon}

where:

$\epsilon$ : a small constant (typically 0.001) to prevent division by zero when the denominator is near zero
$\cos(\vec{d}, \vec{b}) \cdot \cos(\vec{d}, \vec{c})$ : the numerator rewards words with high joint similarity to both $b$ and $c$ . Multiplication means both conditions must be satisfied simultaneously; a word that is highly similar to $b$ but unrelated to $c$ will score poorly.
$\cos(\vec{d}, \vec{a}) + \epsilon$ : the denominator penalizes similarity to $a$ . Words very similar to $a$ are divided out, reducing their score.

The multiplicative structure has an important consequence: the formula requires all cosine similarity values to be non-negative. Cosine similarity ranges from -1 to 1, and negative values in the numerator or denominator would produce undefined or sign-flipped results. The standard implementation shifts all cosine values to the $[0, 1]$ range before applying the formula, using the transformation $\cos' = (\cos + 1) / 2$ .

This shift means 3CosMul and 3CosAdd are not directly comparable on the raw scale: they use different transformations of the same underlying similarity values. But both aim to rank the correct answer highest, and empirically both often succeed.

In[11]:

Code

def three_cos_mul(a, b, c, embeddings, epsilon=0.001, exclude=None):
    """
    Solve analogy a:b :: c:? using 3CosMul.
    Uses multiplicative combination of cosine similarities.
    """
    if exclude is None:
        exclude = {a, b, c}
    scores = {}
    for word, vec in embeddings.items():
        if word not in exclude:
            # Shift cosine to [0, 1] range
            cos_b = (cosine_similarity(vec, embeddings[b]) + 1) / 2
            cos_a = (cosine_similarity(vec, embeddings[a]) + 1) / 2
            cos_c = (cosine_similarity(vec, embeddings[c]) + 1) / 2
            scores[word] = (cos_b * cos_c) / (cos_a + epsilon)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)


results_mul = three_cos_mul("man", "woman", "king", embeddings_2d)

Out[12]:

Console

3CosMul: man is to woman as king is to ?

  1. princess     score=3.0689
  2. queen        score=2.9206
  3. aunt         score=2.5540
  4. prince       score=1.5782
  5. uncle        score=0.5822

When 3CosMul Outperforms 3CosAddLink Copied

The performance difference between 3CosAdd and 3CosMul is subtle and dataset-dependent. Levy et al. (2014) found that 3CosMul performs better on syntactic analogies such as plurals and verb tenses, while 3CosAdd is competitive on semantic analogies such as capitals and currency pairs. The practical recommendation is to try both when evaluating embeddings on a new dataset and to report whichever performs better (or both).

The intuitive explanation for 3CosMul's advantage on syntactic analogies relates to the noise properties of those relationships. Syntactic patterns are encoded through large numbers of morphologically similar examples. The word "dogs" differs from "dog" in ways that affect many contextual words: it appears with plural verbs, with plural determiners, and in contexts where "dog" appears with singular forms. This creates many weak cosine similarity signals in many directions. The additive method's subtraction can amplify these weak signals into noise. The multiplicative method's requirement for joint similarity to both the target and the query word handles these diffuse signals more reliably.

Out[13]:

Visualization

Bar chart showing 3CosAdd scores for each candidate word, with queen scoring highest. — 3CosAdd scores for candidate words in the king:queen analogy. The additive formulation assigns signed scores: positive values indicate words pointing toward the target vector and negative values indicate words pointing away. The correct answer ''queen'' scores highest, while unrelated words receive near-zero or negative scores.

Bar chart showing 3CosMul scores for each candidate word, also with queen highest but different relative score magnitudes. — 3CosMul scores for the same analogy. The multiplicative formulation keeps all scores positive after shifting cosine similarities to [0,1]. The correct answer 'queen' again scores highest, but the relative gap between candidates differs because multiplication penalizes lack of joint similarity more harshly than addition does.

Both methods correctly identify "queen" as the top-ranked answer. The key difference is visible in the score distributions: 3CosAdd produces signed scores where some candidates fall below zero, while 3CosMul keeps all candidates positive. The gap between the top answer and the rest also differs: the multiplicative formulation tends to create sharper distinctions when the correct answer has strong joint similarity to both $b$ and $c$ .