Skip-gram Model › Skip-gram Model

The distributional hypothesis tells us that words appearing in similar contexts carry similar meanings. But turning that observation into something a computer can use requires a representation. Co-occurrence matrices, which we covered in the chapter on classical text representations, capture contextual patterns, yet they are sparse, high-dimensional, and expensive to work with. As vocabulary grows, these matrices balloon in size, and even dimensionality reduction techniques like SVD carry a steep computational price. What if instead of counting how often two words appear together, we trained a model to predict one from the other?

In 2013, Mikolov et al. introduced Word2Vec, a pair of neural network architectures that transformed how we build word representations. The paper appeared in "Efficient Estimation of Word Representations in Vector Space" and, within months of its release, it reshaped the entire field. The Skip-gram model, one of the two Word2Vec designs, takes a deceptively simple approach: given a word, predict the words around it. By training on this prediction task across hundreds of millions of words, the model learns dense vectors that encode rich semantic and syntactic relationships. Words that appear in similar contexts develop similar vector representations. Vector arithmetic starts working: $\vec{\text{king}} - \vec{\text{man}} + \vec{\text{woman}} \approx \vec{\text{queen}}$.

The reason this works is subtle and worth dwelling on before we examine the mechanics. When you train a model to predict context words from a center word, you are forcing the model to compress all the distributional evidence about each word into a fixed-size vector. Two words will only develop similar vectors if they are good at predicting the same context words. "Doctor" and "physician" appear near "hospital," "patient," "diagnosis," and "treatment," so the model learns to give them similar vectors. The geometry of the embedding space is not hand-crafted; it emerges from the statistics of the corpus.

This chapter introduces the Skip-gram architecture from first principles. We will build intuition for why predicting context words forces the model to capture meaning, work through the mathematics step by step, and implement a working model from scratch. By the end, you will understand how Skip-gram works, why each design decision was made, and what trade-offs were accepted in building a practical system.