Word Analogy › What Analogies Reveal About Embeddings

Analogy performance tells you something real about the internal structure of an embedding space, but interpreting that information requires care. A high analogy score doesn't mean the embeddings are good for every task, and a low score doesn't mean they're bad. Understanding what analogies reveal and what they miss is essential for using them as a diagnostic tool rather than treating them as ground truth.

When analogies work well, they reveal three things about an embedding space. First, the space has learned linear structure: relationships are encoded as consistent directional offsets rather than arbitrary transformations. This is a nontrivial property. Nothing in the training objective directly optimizes for linear relationship structure; it emerges as a consequence of how distributional information is compressed into fixed-dimensional vectors. Second, the model has captured distributional regularities: words that appear in similar roles across the corpus cluster together and form coherent subspaces. Third, the training data was rich enough to separate conceptually distinct but co-occurring words into different geometric regions.

When analogies fail, the reasons are instructive and point to real limitations. The parallelogram assumption breaks down for irregular morphological forms. "Go:went" has the same semantic relationship as "run:ran," but "went" and "ran" are not located in the same geometric region relative to "go" and "run" because the surface forms are completely different, and surface form similarity influences embedding geometry. The irregular past tense of "go" has entirely different contextual statistics than the regular pattern. Polysemy creates another failure mode: "bank" as a financial institution and "bank" as a river bank share a single embedding vector, so any analogy involving "bank" is ambiguous. The model must somehow encode two unrelated senses in the same point in space, which guarantees that at least one sense will be poorly located for analogies. And some relationships are not linear: the relationship between cause and effect is complex and multidimensional. You cannot always express the cause of anything as a fixed offset from its effect.

The relationship between training corpus size and analogy performance is important to understand. More training data generally helps, but the gain is uneven across relationship types. Capital-country relationships benefit from large corpora because there are only about 200 countries and their co-occurrence patterns are saturated after seeing a few thousand examples. Irregular verb forms benefit from large corpora because the number of irregular verbs is small and each one needs to be seen in many contexts to build a reliable embedding. By contrast, relationships involving rare entities (obscure currencies, small cities) are fundamentally limited by data sparsity regardless of corpus size.