Word Analogy › Word Analogy and Vector Arithmetic: Semantic Relationships in Embedding Space

Something remarkable happens when you train Skip-gram or CBOW on billions of words: the resulting embeddings exhibit structured geometric relationships that mirror semantic ones. The famous example is $\vec{king} - \vec{man} + \vec{woman} \approx \vec{queen}$. This isn't a coincidence or a numerical trick. The vector from "man" to "woman" captures a gender relationship, and adding it to "king" lands near "queen." The embedding space has learned that kingship relates to queenship the same way manhood relates to womanhood. Royalty and gender are encoded as separate, compositional directions in the same high-dimensional space.

This phenomenon, known as word analogy or vector arithmetic, became one of the most celebrated results in NLP when Mikolov et al. introduced it in 2013 alongside Word2Vec. It was striking evidence that neural embeddings do more than memorize co-occurrence statistics. They organize knowledge into a geometric structure where relationships can be expressed as arithmetic operations. You can reason about language by adding and subtracting vectors.

This chapter explores exactly how and why this works. We develop the parallelogram model that makes analogy completion possible, derive the 3CosAdd and 3CosMul methods for finding answers, examine the standard evaluation datasets researchers use to compare embedding quality, and take an honest look at what analogy performance reveals and where it misleads. By the end, you will understand how to run analogy evaluations, why the technique matters, when it succeeds, and when it fails in ways that matter for real applications.