Word Analogy › Worked Example: Solving Analogies with Synthetic Pre-trained Embeddings
Language AI HandbookAll chapters Word Analogy › Worked Example: Solving Analogies with Synthetic Pre-trained EmbeddingsOne section · Sections in this chapter9 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's see the full analogy evaluation pipeline working end to end. We will create a realistic set of synthetic pre-trained embeddings with explicitly planted relational structure, evaluate both 3CosAdd and 3CosMul, and analyze where each method succeeds and fails.
The synthetic setup is important for a pedagogical reason: with real pre-trained embeddings, the embedding values are fixed and opaque. With synthetic embeddings where we control the relational structure, we can examine exactly what the algorithms find and why.
# uv pip install gensim
import numpy as np
# We'll simulate pre-trained 50-dimensional embeddings for a small vocabulary
# that demonstrates the key analogy properties without requiring a download
def make_embedding_with_offsets(base_vocab, relations, dim=50):
"""
Create synthetic embeddings with planted relational offsets.
relations: list of (group_name, [(word1, word2), ...]) where word2 = word1 + offset
"""
vocab = set(base_vocab)
for _, pairs in relations:
for w1, w2 in pairs:
vocab.add(w1)
vocab.add(w2)
# Random base embeddings
embeddings = {w: np.random.randn(dim) for w in vocab}
# Plant offset directions with noise
for _, pairs in relations:
# Create a consistent offset for this relation
relation_offset = np.random.randn(dim) * 0.5
for w1, w2 in pairs:
embeddings[w2] = (
embeddings[w1] + relation_offset + np.random.randn(dim) * 0.1
)
# Normalize all embeddings
for w in embeddings:
norm = np.linalg.norm(embeddings[w])
if norm > 0:
embeddings[w] = embeddings[w] / norm
return embeddings
# Define relational structure
relations = [
(
"gender",
[
("man", "woman"),
("king", "queen"),
("prince", "princess"),
("actor", "actress"),
("waiter", "waitress"),
],
),
(
"capital",
[
("france", "paris"),
("germany", "berlin"),
("japan", "tokyo"),
("china", "beijing"),
("italy", "rome"),
],
),
(
"plural",
[
("cat", "cats"),
("dog", "dogs"),
("city", "cities"),
("country", "countries"),
("leaf", "leaves"),
],
),
]
base_words = ["the", "a", "this", "that", "is", "was", "are"]
pretrained = make_embedding_with_offsets(base_words, relations, dim=50)We now have a vocabulary of 47 words with three planted relational structures: a gender offset, a capital-city offset, and a plural offset. Each offset is a fixed 50-dimensional vector with small Gaussian noise added per word pair. This mimics the imperfect but consistent structure found in real embeddings.
# Full evaluation with both methods
test_analogies = [
# Gender
("man", "woman", "king", "queen"),
("man", "woman", "prince", "princess"),
("man", "woman", "actor", "actress"),
# Capital cities
("france", "paris", "germany", "berlin"),
("france", "paris", "japan", "tokyo"),
("france", "paris", "italy", "rome"),
# Plural
("cat", "cats", "dog", "dogs"),
("city", "cities", "country", "countries"),
]
add_results = []
mul_results = []
for a, b, c, correct_d in test_analogies:
if all(w in pretrained for w in [a, b, c, correct_d]):
add_top = three_cos_add(a, b, c, pretrained)[0][0]
mul_top = three_cos_mul(a, b, c, pretrained)[0][0]
add_results.append((a, b, c, correct_d, add_top, add_top == correct_d))
mul_results.append((a, b, c, correct_d, mul_top, mul_top == correct_d))
add_accuracy = sum(r[5] for r in add_results) / len(add_results)
mul_accuracy = sum(r[5] for r in mul_results) / len(mul_results)Analogy Answer 3CosAdd 3CosMul -------------------------------------------------------------------- man:woman :: king:? queen OK OK man:woman :: prince:? princess OK OK man:woman :: actor:? actress OK OK france:paris :: germany:? berlin OK OK france:paris :: japan:? tokyo OK OK france:paris :: italy:? rome OK OK cat:cats :: dog:? dogs OK OK city:cities :: country:? countries OK OK 3CosAdd accuracy: 100.0% 3CosMul accuracy: 100.0%
Both methods achieve high accuracy on this synthetic dataset because the relational offsets were explicitly planted and the noise level is controlled. Any errors occur when the noise in the planted offsets is large enough to push the target vector closer to a distractor word than to the correct answer. In real embeddings, this kind of noise is pervasive: the "gender direction" inferred from man/woman is not identical to the direction inferred from king/queen, because each pair accumulates its offset from a different subset of the training corpus.
The key insight for practical use is that this pipeline generalizes directly to real embeddings. Load any pre-trained embeddings (from Gensim, HuggingFace, or your own training), provide analogy triples in the same format, and evaluate both methods to understand which relationship types the embeddings capture. Comparing the category-level accuracy profile across different embedding configurations tells you far more than comparing a single aggregate number.