The Distributional Hypothesis › Evaluating Distributional Models

A legitimate question when building any model is: how do we know if it is working? For distributional models, evaluation typically proceeds along two axes: intrinsic evaluation, which measures the quality of the representations themselves, and extrinsic evaluation, which measures performance on downstream tasks.

The most direct form of intrinsic evaluation compares distributional similarity scores to human judgment datasets. Researchers have collected datasets where pairs of words are rated by many human annotators for their semantic similarity or relatedness, on a scale from 0 (completely unrelated) to 10 (identical meaning). Classic benchmarks include:

• WordSim-353: 353 word pairs rated for similarity and relatedness. The dataset conflates two distinct human judgments (semantic similarity and general relatedness), which makes it somewhat ambiguous but historically important.
• SimLex-999: 999 word pairs with human ratings specifically focused on similarity (words meaning the same thing) rather than association (words that appear together). SimLex-999 is harder for distributional models because it penalizes conflating antonyms with synonyms.
• MEN: 3,000 word pairs rated for semantic relatedness, drawn from a broader vocabulary.
• RG-65: An older but widely-cited dataset of 65 noun pairs with human similarity ratings.

For each dataset, the evaluation computes the correlation (usually Spearman rank correlation) between the distributional cosine similarities and the human ratings. A perfect distributional model would assign high similarity exactly to the pairs humans rate as similar, and low similarity to pairs humans rate as dissimilar.

A standard count-based distributional model achieves Spearman correlations around 0.65 to 0.75 on WordSim-353 after PMI weighting. Neural embedding methods like word2vec and GloVe achieve 0.70 to 0.80. Contextual models like BERT, using representations from the model's final layers, achieve higher scores still. These numbers give a rough calibration: even the best models do not perfectly replicate human similarity judgments, and there is clear headroom for improvement.

The specific choice of Spearman (rank-order) correlation rather than Pearson (linear) correlation is worth explaining. We use Spearman correlation because we care about getting the ranking of word pairs correct, not necessarily the exact numerical values. If a model assigns similarity 0.82 to "cat"/"dog" and 0.41 to "cat"/"bicycle," and humans rate those pairs as 8.5 and 2.1 on a 10-point scale, the ranking is correct even though the absolute values are on different scales. Spearman correlation measures this rank agreement directly.

A second form of intrinsic evaluation uses word analogy tasks, made famous by Mikolov et al.'s 2013 word2vec paper. An analogy task presents a question of the form "man is to woman as king is to ___?" and asks the model to find the word that completes the analogy. The standard approach is to compute:

$$\mathbf{v}_{\text{answer}} \approx \mathbf{v}_{\text{king}} - \mathbf{v}_{\text{man}} + \mathbf{v}_{\text{woman}}$$

and then search for the word in the vocabulary whose vector is nearest to this computed target. If the nearest word is "queen," the analogy is considered solved.

The Google analogy dataset contains 19,544 analogy questions organized into semantic categories (capital cities, country-currency pairs, family relations) and syntactic categories (comparative adjectives, verb tenses, plurals). Word2vec achieves around 65 to 70% accuracy on this dataset. Count-based distributional models typically score lower, around 40 to 55%, motivating the move to neural methods.

Analogy accuracy measures some properties but not others. A model that solves analogies correctly has learned that certain dimensions of the distributional space correspond to systematic semantic transformations (gender, nationality, tense). This is evidence that the distributional representation encodes structured semantic knowledge. But analogy accuracy can be inflated by corpus biases (if the training corpus contains many sentences about capital cities, capital-city analogies will be easy), and it does not directly measure whether the model understands the analogies in a deep sense.

Extrinsic evaluation measures how distributional representations improve performance on downstream NLP tasks when used as features or initialization. Common evaluation tasks include:

• Text classification: Using word vectors as input features and measuring classification accuracy on sentiment analysis, topic classification, or intent detection.
• Named entity recognition: Using word vectors as part of a sequence labeling model and measuring F1 on entity boundaries.
• Parsing: Using word vectors as features in dependency or constituency parsing and measuring labeled attachment scores.
• Question answering: Using word vectors in an end-to-end model and measuring exact match accuracy on reading comprehension benchmarks.

Extrinsic evaluation is more meaningful than intrinsic evaluation for applications: you care whether the representations improve the target task, not whether they correlate with human similarity ratings in the abstract. However, extrinsic evaluation is slower, task-specific, and makes it harder to isolate what property of the representations is responsible for performance differences.