Skip-gram Model › The Embedding Space: Semantic and Syntactic Structure

One of the most striking properties of Skip-gram embeddings, discovered empirically by Mikolov et al., is that meaningful structure emerges in the geometry of the learned space. This goes beyond individual similarities between word pairs. The relationships between words can be computed with vector arithmetic.

The most famous example is:

$$\vec{\text{king}} - \vec{\text{man}} + \vec{\text{woman}} \approx \vec{\text{queen}}$$

The difference vector $\vec{\text{king}} - \vec{\text{man}}$ captures the concept of royalty without gender. Adding $\vec{\text{woman}}$ moves from the "generic person" direction to the "woman" direction. The result lands near "queen" in the embedding space.

This works because of how the embeddings are learned. "King" appears in contexts involving royalty, power, and masculine roles. "Queen" appears in contexts involving royalty, power, and feminine roles. The difference between their embeddings captures the gender axis, not because the model was trained on gender or royalty explicitly, but because these distributional differences emerged from the corpus statistics.

Vector analogies generalize beyond gender. They work for country-capital relationships (France: Paris :: Germany: Berlin), verb tenses (run: ran :: swim: swam), and singular-plural forms (cat: cats :: dog: dogs). The model has implicitly learned grammatical and factual structure, encoded as directions in the embedding space.

It is important to temper this with realistic expectations. The analogy task works impressively for common, high-frequency relationships, but it fails in many edge cases. Less frequent words have noisier embeddings. Polysemous words occupy averaged positions in the space that do not fully capture any single sense. And the analogy test evaluates a narrow property of the embeddings; word similarity benchmarks and downstream task performance are more reliable indicators of embedding quality.

Another way to understand the embedding space is to look at nearest neighbors for specific words. In well-trained Skip-gram embeddings on a large corpus, the nearest neighbors of "Paris" include "London," "Berlin," "Rome," and other European capitals. The nearest neighbors of "running" include "jogging," "sprinting," and "walking." The nearest neighbors of "terrible" include "awful," "horrible," and "dreadful."

These neighborhoods capture different types of relationships depending on window size. Under small windows, "running" might be near "quickly" (because adverbs follow verb phrases) as well as other verbs. Under large windows, "running" might be near "marathon," "race," and "athlete" (because these appear in the same topical contexts).

This is why pretrained embeddings carry implicit choices about the linguistic relationships they capture. When you download a Word2Vec model trained on Google News with a particular configuration, you are implicitly accepting the trade-offs made in window size, dimensionality, and subsampling threshold. Understanding these trade-offs helps you decide whether a given pretrained model is appropriate for your downstream task.