Word Analogy › Summary

Word analogy evaluation reveals how well an embedding space captures semantic and syntactic relationships as linear vector offsets. The central ideas from this chapter are:

• The parallelogram model assumes semantic relationships encode as consistent directional offsets: $\vec{b} - \vec{a} \approx \vec{d} - \vec{c}$ for any two word pairs sharing the same relationship. This structure emerges naturally from distributional training because words in analogous roles appear in analogous contexts.

• Vector arithmetic predicts the answer word by computing $\vec{b} - \vec{a} + \vec{c}$ and finding the nearest vocabulary word. The algebraic simplicity of this operation is one of the most striking properties of well-trained word embeddings.

• 3CosAdd searches for the word maximizing cosine similarity to the target vector, equivalently written as $\cos(\vec{d}, \vec{b}) - \cos(\vec{d}, \vec{a}) + \cos(\vec{d}, \vec{c})$. The explicit form reveals the selection criterion: find a word that resembles $b$, resembles $c$, but does not resemble $a$.

• 3CosMul uses a multiplicative formulation $\frac{\cos(\vec{d}, \vec{b}) \cdot \cos(\vec{d}, \vec{c})}{\cos(\vec{d}, \vec{a}) + \epsilon}$ that avoids negative scores and is less sensitive when cosine similarities are near zero. It performs better on syntactic analogies.

• Cosine similarity is preferred over Euclidean distance because it is magnitude-invariant, measuring directional alignment rather than vector length differences.

• The Google Analogy Dataset, MSR Syntactic Dataset, and BATS provide standardized evaluation across semantic and syntactic relationship types, with BATS offering the finest diagnostic granularity.

• Analogy accuracy correlates weakly with downstream task performance, so analogy evaluation is best used as a fast development diagnostic rather than a primary quality measure. Optimizing for analogy accuracy does not reliably improve performance on classification, parsing, or generation tasks.

• Analogies fail on irregular morphology, polysemous words, and relationships that aren't linearly encodable. High accuracy indicates the embedding space has linear relational structure for the tested categories; it does not imply general linguistic competence.

• Analogy datasets encode cultural biases, so high accuracy on biased analogy pairs indicates the model has absorbed those biases from training data, not that it has learned neutral linguistic representations.

In the next chapter, we explore GloVe, which takes a fundamentally different approach to learning embeddings: instead of predicting context words one at a time, it directly factorizes the global co-occurrence matrix built from the entire corpus. GloVe's design will illuminate why both predictive and count-based methods can achieve similar analogy performance despite their very different training objectives, and what that convergence tells us about the information content of distributional statistics.