Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Word Analogy › The Parallelogram Model

Michael BrenndoerferApril 5, 20259 min read

Language AI HandbookAll chapters

Part 4 of 67Word Embeddings

Chapter 24Word Analogy › The Parallelogram Model

Reading section 2 of 13 · The Parallelogram Model · 9 minSection 2 of 13 · 9 min

Section 2 of 13 · 9 min read

Word Analogy › The Parallelogram ModelOne section · Sections in this chapter2 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Word analogies rest on a geometric intuition: if two word pairs share the same semantic relationship, the vectors connecting them should be parallel and of similar length. This is the parallelogram model, and it is the foundation for all analogy-based evaluation.

Consider the relationship "capital city of." Paris is the capital of France, Tokyo is the capital of Japan, and Berlin is the capital of Germany. If embeddings capture this relationship consistently, then the vector pointing from France to Paris should be roughly the same as the vector from Japan to Tokyo and from Germany to Berlin. Each of those difference vectors encodes the same directional displacement: "country to its capital city." This directional consistency is what makes analogy completion possible. If you know three of the four words in a pair of analogous relationships, you can predict the fourth by applying the relationship vector.

To see why this structure would arise during training, think about how Skip-gram learns. Words that appear in similar contexts end up with similar embeddings. France, Germany, and Japan all appear in contexts like "the prime minister of ___", "the economy of ___", "traveled to ". Paris, Tokyo, and Berlin appear in contexts like "flew into ___ this morning", " is a vibrant city", "the Eiffel Tower in ___". The contextual patterns for all three countries are similar to each other, and the contextual patterns for all three capitals are similar to each other. The model pulls France, Germany, and Japan into a cluster, and Paris, Tokyo, and Berlin into another cluster. The geometric displacement from one cluster to the other encodes the capital-of relationship. When this happens for dozens of country-capital pairs, the displacement vectors become aligned.

Parallelogram Model

The parallelogram model assumes semantic relationships are encoded as consistent vector offsets. If words $a$ and $b$ share a relationship $R$ (for example, "man" and "woman" share a gender relationship), and words $c$ and $d$ share the same relationship (for example, "king" and "queen"), then the offset vectors should be approximately equal: $\vec{b} - \vec{a} \approx \vec{d} - \vec{c}$ . Geometrically, this means the four word vectors form a parallelogram in embedding space, where the two relationship vectors are parallel sides.

The model is called a parallelogram because of the shape formed when you draw the four word vectors as points in space and connect them with the relationship arrows. The two "relationship" edges are parallel: $\vec{man} \to \vec{woman}$ runs parallel to $\vec{king} \to \vec{queen}$ . The two "word class" edges are also parallel: $\vec{man} \to \vec{king}$ runs parallel to $\vec{woman} \to \vec{queen}$ . This is the geometric definition of a parallelogram, and it is the structure we are detecting when we evaluate analogies.

In[3]:

Code

import numpy as np

# Simulate embeddings that exhibit parallelogram structure
# In real embeddings, this structure emerges from training

# Create a 2D example for visualization
# The "gender" direction is a fixed offset vector
gender_offset = np.array([0.8, 0.3])

# Base embeddings for male terms
man = np.array([-0.5, 0.2])
king = np.array([0.3, 0.9])
prince = np.array([0.1, 0.6])
uncle = np.array([-0.3, 0.4])

# Female terms: same as male + gender offset
woman = man + gender_offset
queen = king + gender_offset
princess = prince + gender_offset
aunt = uncle + gender_offset

embeddings_2d = {
    "man": man,
    "woman": woman,
    "king": king,
    "queen": queen,
    "prince": prince,
    "princess": princess,
    "uncle": uncle,
    "aunt": aunt,
}

Out[4]:

Visualization

2D scatter plot showing word embeddings with parallel purple arrows between male and female word pairs forming parallelograms. — The parallelogram model of word analogies in 2D embedding space. Each pair of gendered terms (man/woman, king/queen, prince/princess, uncle/aunt) is connected by the same purple offset vector. This consistent directional offset allows analogy completion: starting from ''king'' and applying the man-to-woman gender direction lands precisely at ''queen''. When real embeddings are trained on large corpora, this parallelogram structure emerges automatically because words in analogous roles appear in analogous contexts.

The parallelogram structure shown above is idealized, with a perfectly consistent gender offset. Real embeddings have noise: the offsets aren't perfectly parallel because each word is trained from noisy co-occurrence statistics in natural text. "King" appears in contexts that "prince" does not, and "man" appears in everyday contexts that "uncle" does not. So the offsets won't be identical. But when embeddings are trained well on large corpora, the relationship vectors are similar enough in direction that analogy completion works reliably for many relationship types. The key insight is that it doesn't need to be perfect. You just need the correct answer to be closer to the target than any other word in the vocabulary.

Why Linear Structure Emerges From Distributional TrainingLink Copied

It is worth pausing to understand why this linear structure should emerge at all from a model that is simply trained to predict context words. The answer connects distributional semantics to linear algebra in a way that illuminates why the analogy trick works.

When Skip-gram trains on a sentence like "The king ruled his kingdom wisely," it updates the embedding for "king" to better predict the surrounding words: "the," "ruled," "his," "kingdom," "wisely." The same word "ruled" also appears around other powerful entities: generals, prime ministers, emperors. So "general," "king," and "emperor" end up with overlapping context distributions. They get pulled toward similar embedding regions.

Now consider gender. Every word that participates in a gender-paired relationship (man/woman, king/queen, actor/actress, waiter/waitress) appears in two kinds of contexts: its own semantic contexts (royalty, service, performance) and gender-indicative contexts. "Queen" appears with "she," "her," "her majesty." "King" appears with "he," "his," "his majesty." The model learns a representation for "queen" that is pulled toward the "royalty" region (because of shared semantic contexts with "king") and simultaneously displaced in a direction associated with feminine pronouns and references (because of gender-indicative context words). This displacement is approximately the same for all gender pairs, because they all get pushed in the same direction by the same set of gendered context words.

The result is a shared directional offset: a gender axis in the embedding space. The position along this axis distinguishes male from female terms; the position in the orthogonal subspace distinguishes the semantic category (royal vs. occupational vs. familial). This decomposition of meaning into compositional subspaces is the deep reason analogy arithmetic works.

Vector Arithmetic for AnalogiesLink Copied

The classic analogy task asks: " $a$ is to $b$ as $c$ is to ?". If the relationship between $a$ and $b$ is the same as the relationship between $c$ and $d$ , then the offset vectors should be approximately equal: $\vec{b} - \vec{a} \approx \vec{d} - \vec{c}$ . Rearranging this equation gives us the target vector we are searching for:

\vec{d} \approx \vec{b} - \vec{a} + \vec{c}

where:

$\vec{a}$ , $\vec{b}$ , $\vec{c}$ , $\vec{d}$ : the embedding vectors for the respective words
$\vec{b} - \vec{a}$ : the relationship vector capturing how $b$ differs from $a$
$\vec{c}$ : the starting point from which we apply the relationship

The process has three conceptually distinct steps. First, compute the relationship vector $\vec{b} - \vec{a}$ , which captures the directional change associated with the relationship. For the gender analogy, this is the vector pointing from "man" to "woman," representing the direction of femininity in embedding space. Second, apply this offset to $\vec{c}$ to predict where $\vec{d}$ should be. Adding the gender offset to "king" produces a point that should be near the female-equivalent-of-king, which is "queen." Third, search the vocabulary for the word whose embedding is closest to this computed target vector. The word with the nearest embedding is the predicted answer.

The simplicity of this process is remarkable. Three vector additions and a nearest-neighbor search are enough to solve thousands of semantic and syntactic analogy questions.

In[5]:

Code

def analogy_vector(a, b, c, embeddings):
    """Compute the target vector for analogy: a is to b as c is to ?"""
    return embeddings[b] - embeddings[a] + embeddings[c]


# Example: man is to woman as king is to ?
target = analogy_vector("man", "woman", "king", embeddings_2d)

Out[6]:

Console

Analogy: man is to woman as king is to ?
--------------------------------------------------
vec(man):   [-0.5  0.2]
vec(woman): [0.3 0.5]
vec(king):  [0.3 0.9]

Relationship (woman - man): [0.8 0.3]
Target vector:              [1.1 1.2]
Actual queen:               [1.1 1.2]

Distance between target and queen: 0.000000

The distance of zero confirms that our synthetic embeddings exhibit perfect parallelogram structure: the computed target vector exactly matches "queen." In real embeddings there is noise, so we must search the vocabulary for the nearest word by some distance metric, which brings us to the question of which metric to use.