Word Analogy › Limitations of Analogy Evaluation

Analogy accuracy has become a standard benchmark, but its limitations are substantial enough to warrant careful interpretation. Using it as the primary measure of embedding quality can mislead research and development in subtle but important ways.

The 3CosAdd evaluation excludes the three query words $\{a, b, c\}$ from the candidate set. This prevents trivially returning one of the input words as the answer. For example, without exclusion, "woman" would be a valid response to "man:woman::king:?" since "woman" is already present as the source of the relationship vector. The exclusion is clearly necessary for a meaningful evaluation.

But the exclusion creates a subtler artifact. In many cases, the correct answer word lies very close to one of the excluded input words in embedding space. "Queen" is close to "king" by virtue of being in the royalty cluster. Without exclusion, "king" might be the nearest neighbor to the target vector. With exclusion, the evaluation forces the algorithm to find the second or third nearest neighbor, which happens to be "queen." This means the algorithm is being evaluated partly on its ability to move away from the input words rather than purely on its ability to find the correct answer by relationship structure. Removing the exclusion significantly reduces apparent accuracy, suggesting that a portion of "correct" answers benefit from this artificial constraint.

Analogy accuracy depends simultaneously on vocabulary coverage, embedding dimension, training corpus size, preprocessing choices, and the evaluation method. Comparing accuracy across models is valid only when these factors are controlled. A 300-dimensional model trained on 100 billion tokens should not be directly compared to a 50-dimensional model trained on 1 billion tokens. The differences in accuracy reflect the training conditions as much as the model architecture.

This confounding is not a minor inconvenience. In the original Word2Vec paper, Mikolov et al. reported analogy accuracy as evidence for the superiority of their model over baselines. But the comparison used different training data sizes, making it impossible to attribute the improvement to the model architecture rather than the larger corpus. Subsequent controlled experiments have found that corpus size, preprocessing, and hyperparameter choices explain a large fraction of the performance differences between Word2Vec, GloVe, and other methods.

The deeper problem is that analogy accuracy measures a property of embedding space geometry, not performance on downstream tasks. A model might achieve high analogy accuracy while performing poorly on text classification, named entity recognition, or question answering. The connection between analogy accuracy and task performance is empirically weak.

Faruqui et al. (2016) systematically examined the correlation between intrinsic embedding evaluations, including analogy accuracy and word similarity judgments, and performance on downstream tasks. They found low and inconsistent correlations across tasks and datasets. For some tasks, higher intrinsic scores were associated with slightly higher downstream scores. For others, there was no meaningful relationship. This finding has been replicated and extended multiple times. The message is clear: intrinsic benchmarks like analogy accuracy are not reliable proxies for downstream task performance.

Why might a model with high analogy accuracy perform poorly on downstream tasks? One reason is task mismatch. The analogy dataset emphasizes encyclopedic and morphological relationships that appear frequently in Wikipedia-style text. A model trained on that distribution will learn those relationship types well, but might perform poorly on sentiment analysis (where emotional associations matter), medical NLP (where domain-specific terminology determines performance), or social media text classification (where informal language dominates). The analogy benchmark is not representative of the full space of language use.

Another reason is the nature of linear structure itself. High analogy accuracy indicates the model has learned linear subspaces for specific relationship types. But many downstream tasks require nonlinear reasoning, multi-word compositions, or document-level understanding that simple embedding lookups cannot provide. A model optimized for linear analogy accuracy is optimized for a very specific geometric property that may not transfer to tasks requiring richer representations.

The analogy datasets encode cultural and demographic biases that become embedded in the evaluation itself. The Google Analogy Dataset was constructed by manually selecting word pairs, and this selection reflects the perspectives and assumptions of its creators.

Family relationship analogies use binary gender categories: man:woman, brother:sister, king:queen. This reflects a particular cultural framing of gender as a binary categorical variable encoded in language, which may not generalize across languages, time periods, or communities. Currency pair analogies favor widely-known currencies, biasing toward wealthy and prominent nations. Many currencies of smaller countries are not represented.

The most pointed example is occupational analogies. If a dataset includes analogies like "man:doctor::woman:nurse," evaluating on that dataset rewards models that have learned occupational gender stereotypes. High accuracy on such questions indicates the model has absorbed stereotypical associations from its training data, not that it has learned linguistically correct or socially neutral representations. This confounds two very different kinds of knowledge: linguistic structure and cultural stereotypes.

This is not a reason to abandon analogy evaluation entirely. It is a reason to interpret analogy accuracy as evidence about what patterns the training data contained, not as evidence about neutral linguistic competence. A model that scores well on biased analogies has learned the biases in the corpus, which may be useful to know. A model that scores poorly on biased analogies might have undergone debiasing, or it might simply lack coverage. The accuracy number alone cannot distinguish these cases.

A model achieving 75% analogy accuracy has demonstrably learned linear subspaces corresponding to certain semantic and syntactic relationships. This is meaningful evidence. It indicates the model has captured some distributed structure from the training data, and that this structure is geometrically organized in a way that allows relationship-based reasoning. The fact that "king - man + woman" lands near "queen" tells you something real about the embedding geometry.

But it doesn't mean the embeddings represent language understanding in a general sense. It doesn't mean they will transfer to all tasks. It doesn't mean they are unbiased. And it doesn't mean a model that achieves 80% accuracy is better for your application than one that achieves 70%. Analogy evaluation is one lens among many, and it is most useful when combined with extrinsic evaluations on real tasks, similarity benchmarks on human judgment data, and qualitative analysis of the embeddings themselves.