Skip-gram Model › The Core Idea: Predicting Context from Words
Language AI HandbookAll chapters Skip-gram Model › The Core Idea: Predicting Context from WordsOne section · Sections in this chapter2 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Most early word representation methods built their representations by counting. Count how often "bank" appears near "river," how often it appears near "money," and use those counts as features. The resulting vectors work, but they suffer from sparsity and scale poorly to large vocabularies. A vocabulary of 100,000 words requires a co-occurrence matrix with 10 billion entries. Most of those entries are zero, because most pairs of words never appear near each other in any corpus. Sparse representations are expensive to store, slow to manipulate, and difficult to generalize from.
Skip-gram takes a different angle. Instead of storing counts, it trains a model to predict: given the word "fox," what words tend to appear nearby? The model is never asked to identify what "fox" means in the abstract. It is simply asked to predict context. But in learning to predict context well, it discovers which words share contextual patterns, and those shared patterns encode meaning.
The Skip-gram model learns word representations by training a neural network to predict context words given a center word. The learned weights of this network become the word embeddings.
The training setup is built around sliding windows. You take a sentence, pick a center word, and look at a fixed window around it. Every word in that window becomes a target the model should predict given the center word. This process is repeated for every word in the corpus, sweeping the window from left to right across each sentence.
# Demonstrating Skip-gram's training data generation
sentence = "The quick brown fox jumps over the lazy dog"
words = sentence.lower().split()
def get_skipgram_pairs(words, target_idx, window_size=2):
"""Generate (center, context) pairs for Skip-gram training."""
center = words[target_idx]
pairs = []
for offset in range(-window_size, window_size + 1):
if offset == 0:
continue
context_idx = target_idx + offset
if 0 <= context_idx < len(words):
pairs.append((center, words[context_idx]))
return pairs
# Generate all training pairs from the sentence
all_pairs = []
for i in range(len(words)):
all_pairs.extend(get_skipgram_pairs(words, i, window_size=2))Skip-gram Training Pairs (window size = 2) -------------------------------------------------- Sentence: 'The quick brown fox jumps over the lazy dog' Sample pairs for 'fox' (index 3): 'fox' -> 'quick' 'fox' -> 'brown' 'fox' -> 'jumps' 'fox' -> 'over' Total training pairs from this sentence: 30
Notice that a single center word generates multiple training pairs, one for each context position. A single occurrence of "fox" in a sentence with window size 2 produces four training pairs. This is an important design property: the model gets many learning signals from each word occurrence, all tied to the same center-word vector. Every gradient update that makes "fox" better at predicting "brown" also shapes the same vector that must predict "jumps." The final embedding for "fox" must simultaneously satisfy all these constraints, which forces it to encode the full distributional character of the word.
The diagram below shows how a sliding window sweeps across a sentence, generating (center, context) pairs at each position. As the window moves, every word takes a turn as the center, and its neighbors become prediction targets.
Window Size: A Key Hyperparameter
The window size controls how far left and right from the center word we look for context. This choice has a deeper effect than it might appear, because it determines what kind of linguistic relationship the model is trained to capture.
Smaller windows (size 2) force the model to predict words that are grammatically related to the center word. Adjectives appear right before their nouns, objects appear right after their verbs, and determiners immediately precede the nouns they modify. By predicting these close neighbors, the model learns embeddings that reflect syntactic roles and relationships. Two words with similar syntactic functions, like "big" and "small" or "run" and "walk," will share similar close neighbors and therefore develop similar embeddings under a small window.
Larger windows (size 5 to 10) capture broader co-occurrences. Within a 10-word window, words that discuss the same topic but are not grammatically linked will appear together. In a paragraph about cooking, "recipe," "ingredients," "oven," and "temperature" may not be grammatical neighbors, but they are topical neighbors. Skip-gram with a large window learns embeddings that reflect topical or semantic similarity rather than syntactic similarity. The word "doctor" will be near "hospital" and "nurse" in semantic space, even though these words rarely appear as direct grammatical partners.
This trade-off has practical consequences for downstream tasks. Embeddings trained with small windows work better for syntactic tasks like part-of-speech tagging and parsing. Embeddings trained with large windows work better for semantic tasks like word similarity, analogy solving, and document classification. When you use a pretrained Word2Vec model, checking the window size used during training tells you something important about what kind of relationships the embeddings capture. The original Word2Vec paper used window sizes of 5 to 10 for most experiments, aiming for broad semantic coverage.
There is also a subtler consideration around word frequency. In any corpus, high-frequency words like "the," "a," and "is" appear constantly, and with a large window they become context words for almost everything. This can overwhelm the training signal. Mikolov et al. introduced subsampling to address this: frequent words are randomly discarded during training data generation with a probability that scales with their frequency. This prevents the most common words from dominating the gradient signal and allows rarer, more informative words to contribute more to the learning process.
Sentence Boundaries and Context
One important implementation detail is that the sliding window should not cross sentence boundaries. If you naively apply a window of size 5 at the end of one sentence and the beginning of the next, you will generate spurious (center, context) pairs from words that are unrelated. Real implementations either split the corpus into sentences first or use special sentence-boundary tokens. In practice, the effect is modest for large corpora but matters when you want clean, interpretable embeddings.