Skip-gram Model › Summary

Skip-gram learns word embeddings by training a shallow neural network to predict context words from a center word. The key ideas are:

• Training signal: Every word occurrence generates multiple (center, context) training pairs via a sliding window. No manual labels are required; the corpus provides its own supervision through the distributional structure of language.
• Architecture: Two embedding matrices (center embeddings $\mathbf{W}$ and context embeddings $\mathbf{W}'$) are learned jointly. After training, the center embeddings serve as the word representations, with each row corresponding to one word's $d$-dimensional vector.
• Objective: Maximize the log-likelihood of observed (center, context) pairs using softmax over the full vocabulary. This pulls together embeddings of words that co-occur and pushes apart embeddings of words that do not.
• Window size: A critical hyperparameter controlling whether the model captures syntactic (small window) or semantic (large window) relationships. The choice shapes the kind of similarity the embeddings encode.
• Dual representations: Every word has a center embedding (used when predicting context) and a context embedding (used when being predicted). This asymmetry allows the model to capture directional co-occurrence patterns.
• Skip-gram vs. CBOW: Skip-gram makes one prediction per context position, generating more learning signals per occurrence and benefiting rare words. CBOW averages context embeddings and makes one prediction per center word, training faster but with less benefit for rare vocabulary.
• Subsampling: Discarding frequent words during training data generation reduces noise and gives rarer, more informative words greater influence on the embeddings.

The full softmax objective described here is exact but computationally expensive. The next chapters cover Negative Sampling and Hierarchical Softmax, which make Skip-gram tractable at the scale of real-world corpora while preserving the essential learning dynamics.