Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Embedding Evaluation › Word Similarity Datasets

Michael BrenndoerferUpdated March 21, 20269 min read

Language AI HandbookAll chapters

Part 4 of 67Word Embeddings

Chapter 27Embedding Evaluation › Word Similarity Datasets

Reading section 3 of 13 · Word Similarity Datasets · 9 minSection 3 of 13 · 9 min

Section 3 of 13 · 9 min read

Embedding Evaluation › Word Similarity DatasetsOne section · Sections in this chapter3 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The oldest and most widely used intrinsic evaluation method asks humans to rate how similar pairs of words are, then checks whether embedding cosine similarities correlate with those human judgments. The method goes back to 1965 and remains foundational today because human similarity judgments are a natural proxy for what we want from semantic representations.

How Similarity Evaluation WorksLink Copied

The procedure starts with a dataset of word pairs, each annotated with a human similarity score. Annotators see pairs like ("cat", "kitten") or ("car", "fuel") and assign a score on a fixed scale, typically 0 to 10, where 0 means completely unrelated and 10 means identical in meaning. These judgments are then averaged across annotators to produce a consensus score for each pair.

Given a dataset of word pairs with human similarity scores, the evaluation procedure is:

For each word pair $(w_1, w_2)$ , retrieve the embeddings $\mathbf{v}_1$ and $\mathbf{v}_2$
Compute cosine similarity between the two embeddings:

\text{sim}(w_1, w_2) = \frac{\mathbf{v}_1 \cdot \mathbf{v}_2}{|\mathbf{v}_1| \, |\mathbf{v}_2|}

where $\mathbf{v}_1 \cdot \mathbf{v}_2$ is the dot product of the two vectors and $|\mathbf{v}_1|$ , $|\mathbf{v}_2|$ are their Euclidean norms. Cosine similarity measures the angle between two vectors rather than their magnitude. It ranges from $-1$ (opposite directions) to $1$ (identical directions), with $0$ indicating orthogonal (unrelated) vectors. Using cosine rather than Euclidean distance is important because word vectors often vary in magnitude for reasons unrelated to semantic similarity, and dividing by the norms removes this confound.

Compute Spearman rank correlation $\rho$ between the list of embedding similarities and the corresponding human scores
Report $\rho$ as the evaluation score

Why Spearman Rank CorrelationLink Copied

Spearman correlation is preferred over Pearson because it measures rank agreement rather than linear correlation. Human similarity ratings are ordinal data: the gap between a score of 4 and a score of 5 may not represent the same psychological distance as the gap between 7 and 8. Annotators don't calibrate a linear scale; they produce rankings. Spearman correlation respects this by comparing the relative ordering of pairs rather than the absolute numeric differences.

There's a second reason to prefer Spearman: embedding cosine similarities don't necessarily align linearly with human similarity judgments even when the rank order is correct. The human scale runs from 0 to 10 with a roughly uniform prior. Cosine similarities cluster near 0 for most pairs and near 1 only for very similar words. This compression means linear correlation (Pearson) would be systematically poor even for good embeddings. Spearman sidesteps this by working with ranks.

Major Similarity DatasetsLink Copied

Several benchmark datasets have become standards in the field, each designed to test different aspects of semantic representation:

WordSim-353 (Finkelstein et al., 2001): 353 English word pairs with human similarity judgments on a 0-10 scale. One of the earliest and still widely used benchmarks. It conflates two distinct notions: semantic similarity ("car"/"automobile") and semantic relatedness ("car"/"gas"). This conflation turned out to be a significant issue because models can score well on it by capturing either relatedness or similarity.
SimLex-999 (Hill et al., 2015): 999 word pairs specifically designed to measure semantic similarity, not relatedness. The creators explicitly told annotators to rate similarity and not association, with careful instructions distinguishing the two. "Cat" and "dog" are related but not similar; "cat" and "kitten" are both related and similar. SimLex is generally considered a harder and more meaningful benchmark than WordSim, and a model's SimLex score is a better indicator of whether it has learned true semantic equivalence.
MEN (Bruni et al., 2014): 3000 word pairs covering both content and function words, with relatedness judgments collected via crowdsourcing. Useful for testing coverage across word types. The larger size makes correlation estimates more statistically reliable.
RG-65 (Rubenstein and Goodenough, 1965): The granddaddy of similarity datasets, containing just 65 carefully chosen noun pairs. Small but historically important as the benchmark that established the methodology and is still used to compare modern models to results from sixty years ago.
SimVerb-3500 (Gerz et al., 2016): 3500 verb pairs with similarity judgments. Verbs are harder to evaluate than nouns because verbal similarity is more context-dependent: "run" and "sprint" are similar in some senses but "run" and "manage" (as in "run a company") might also be considered similar in others.

Running evaluation on multiple datasets simultaneously is good practice because each captures a different aspect of semantic quality. A model might score well on MEN (relatedness) but poorly on SimLex (true similarity), which tells you something specific about what properties the model has and hasn't learned.

Let's implement word similarity evaluation from scratch:

In[3]:

Code

import numpy as np

# Simulate embeddings using random vectors with structure
# (In practice, you'd load real GloVe or Word2Vec embeddings)
np.random.seed(42)


def make_structured_embeddings(dim=50):
    """Create embeddings with semantic structure for demonstration."""
    embeddings = {}

    # Animal cluster
    animal_base = np.random.randn(dim)
    embeddings["cat"] = animal_base + 0.2 * np.random.randn(dim)
    embeddings["kitten"] = animal_base + 0.15 * np.random.randn(dim)
    embeddings["dog"] = animal_base + 0.4 * np.random.randn(dim)
    embeddings["puppy"] = animal_base + 0.35 * np.random.randn(dim)
    embeddings["horse"] = animal_base + 0.7 * np.random.randn(dim)

    # Vehicle cluster
    vehicle_base = np.random.randn(dim) * 2
    embeddings["car"] = vehicle_base + 0.2 * np.random.randn(dim)
    embeddings["automobile"] = vehicle_base + 0.1 * np.random.randn(dim)
    embeddings["truck"] = vehicle_base + 0.5 * np.random.randn(dim)
    embeddings["bike"] = vehicle_base + 0.8 * np.random.randn(dim)
    embeddings["bus"] = vehicle_base + 0.6 * np.random.randn(dim)

    # Technology cluster
    tech_base = np.random.randn(dim) * 3
    embeddings["computer"] = tech_base + 0.2 * np.random.randn(dim)
    embeddings["laptop"] = tech_base + 0.3 * np.random.randn(dim)
    embeddings["phone"] = tech_base + 0.7 * np.random.randn(dim)
    embeddings["software"] = tech_base + 0.5 * np.random.randn(dim)
    embeddings["program"] = tech_base + 0.4 * np.random.randn(dim)

    # Normalize all embeddings
    for word in embeddings:
        embeddings[word] = embeddings[word] / np.linalg.norm(embeddings[word])

    return embeddings


embeddings = make_structured_embeddings()

In[4]:

Code

def cosine_similarity(v1, v2):
    """Compute cosine similarity between two vectors."""
    return np.dot(v1, v2) / (np.linalg.norm(v1) * np.linalg.norm(v2))


# Simulated word similarity dataset (word1, word2, human_score)
# Scores range from 0 (unrelated) to 10 (identical meaning)
similarity_dataset = [
    ("cat", "kitten", 9.0),  # Very similar
    ("dog", "puppy", 8.5),  # Very similar
    ("cat", "dog", 6.0),  # Related but not identical
    ("car", "automobile", 9.5),  # Near synonyms
    ("car", "truck", 5.5),  # Same category
    ("car", "bike", 4.5),  # Loosely related
    ("car", "cat", 1.5),  # Unrelated
    ("computer", "laptop", 8.0),  # Very similar
    ("computer", "phone", 5.0),  # Related
    ("horse", "car", 1.0),  # Unrelated
    ("software", "program", 7.5),  # Similar
    ("bus", "truck", 5.0),  # Same category
]

In[5]:

Code

from scipy.stats import spearmanr


def evaluate_word_similarity(embeddings, dataset):
    """
    Evaluate word embeddings on a similarity dataset.
    Returns Spearman correlation with human judgments.
    """
    embedding_scores = []
    human_scores = []
    skipped = []

    for word1, word2, human_score in dataset:
        if word1 not in embeddings or word2 not in embeddings:
            skipped.append((word1, word2))
            continue

        sim = cosine_similarity(embeddings[word1], embeddings[word2])
        embedding_scores.append(sim)
        human_scores.append(human_score)

    correlation, p_value = spearmanr(embedding_scores, human_scores)

    return {
        "correlation": correlation,
        "p_value": p_value,
        "n_pairs": len(embedding_scores),
        "n_skipped": len(skipped),
        "skipped": skipped,
        "details": list(
            zip(
                [
                    d[0]
                    for d in dataset
                    if d[0] in embeddings and d[1] in embeddings
                ],
                [
                    d[1]
                    for d in dataset
                    if d[0] in embeddings and d[1] in embeddings
                ],
                embedding_scores,
                human_scores,
            )
        ),
    }


results = evaluate_word_similarity(embeddings, similarity_dataset)

Out[6]:

Console

Word Similarity Evaluation Results
=======================================================
Pairs evaluated:  12
Pairs skipped:    0
Spearman rho:     0.6480
P-value:          0.0227

Word 1       Word 2        Emb Sim    Human
---------------------------------------------
cat          kitten          0.966      9.0
dog          puppy           0.885      8.5
cat          dog             0.915      6.0
car          automobile      0.989      9.5
car          truck           0.956      5.5
car          bike            0.893      4.5
car          cat            -0.242      1.5
computer     laptop          0.995      8.0
computer     phone           0.982      5.0
horse        car            -0.165      1.0
software     program         0.984      7.5
bus          truck           0.892      5.0

The Spearman correlation tells you how well your embedding similarities rank-agree with human judgments. A correlation of 0.75 means the embeddings roughly agree with human intuitions about which pairs are most similar. Perfect agreement would be 1.0, and state-of-the-art models on SimLex-999 typically achieve around 0.75-0.85. Human inter-annotator agreement on SimLex is about 0.67, which provides an important reference point: no model can be expected to exceed the ceiling set by human agreement.

Interpreting Correlation Scores in ContextLink Copied

Reading a single Spearman correlation number is necessary but not sufficient. Several contextual factors shape what the number means. First, statistical significance matters: a correlation of 0.70 on a dataset of 65 pairs (RG-65) is much less reliable than the same correlation on 999 pairs (SimLex-999). Compute the p-value and check that it's below a meaningful threshold, typically 0.01 for word similarity benchmarks.

Second, comparisons across datasets are not apples-to-apples. WordSim-353 and SimLex-999 measure different things, and a model's relative ranking on the two datasets can be quite different. Report results on multiple benchmarks whenever possible. Third, the correlation is computed only over pairs where both words are in the vocabulary. If you skip many pairs, the reported correlation is on an unrepresentative sample.

Coverage IssuesLink Copied

A critical practical issue is that embeddings may not contain all words in the evaluation dataset. When a word is missing, you either skip the pair or handle it specially (with a random vector or zero vector, for example). The choice matters: always report what fraction of pairs were skipped. Embedding vocabularies trained on one corpus often miss rare or specialized words that appear in evaluation datasets.

The OOV problem interacts with score interpretation in a subtle way. Rare words tend to have noisier, less reliable embeddings when they do appear in the vocabulary, and they're the most likely to be missing when they don't. If your vocabulary covers only common words, you miss the pairs that would be hardest to evaluate correctly, which means your reported correlation is computed on an easier subset than the full benchmark. A model that covers 95% of vocabulary and achieves 0.70 correlation is almost certainly better than one that covers 60% and achieves 0.75 correlation on the pairs it can handle.