Skip-gram Model: Word2Vec Architecture and Word Embeddings

The distributional hypothesis tells us that words appearing in similar contexts carry similar meanings. But turning that observation into something a computer can use requires a representation. Co-occurrence matrices, which we covered in the chapter on classical text representations, capture contextual patterns, yet they are sparse, high-dimensional, and expensive to work with. As vocabulary grows, these matrices balloon in size, and even dimensionality reduction techniques like SVD carry a steep computational price. What if instead of counting how often two words appear together, we trained a model to predict one from the other?

In 2013, Mikolov et al. introduced Word2Vec, a pair of neural network architectures that transformed how we build word representations. The paper appeared in "Efficient Estimation of Word Representations in Vector Space" and, within months of its release, it reshaped the entire field. The Skip-gram model, one of the two Word2Vec designs, takes a deceptively simple approach: given a word, predict the words around it. By training on this prediction task across hundreds of millions of words, the model learns dense vectors that encode rich semantic and syntactic relationships. Words that appear in similar contexts develop similar vector representations. Vector arithmetic starts working: $\vec{\text{king}} - \vec{\text{man}} + \vec{\text{woman}} \approx \vec{\text{queen}}$.

The reason this works is subtle and worth dwelling on before we examine the mechanics. When you train a model to predict context words from a center word, you are forcing the model to compress all the distributional evidence about each word into a fixed-size vector. Two words will only develop similar vectors if they are good at predicting the same context words. "Doctor" and "physician" appear near "hospital," "patient," "diagnosis," and "treatment," so the model learns to give them similar vectors. The geometry of the embedding space is not hand-crafted; it emerges from the statistics of the corpus.

This chapter introduces the Skip-gram architecture from first principles. We will build intuition for why predicting context words forces the model to capture meaning, work through the mathematics step by step, and implement a working model from scratch. By the end, you will understand how Skip-gram works, why each design decision was made, and what trade-offs were accepted in building a practical system.

Most early word representation methods built their representations by counting. Count how often "bank" appears near "river," how often it appears near "money," and use those counts as features. The resulting vectors work, but they suffer from sparsity and scale poorly to large vocabularies. A vocabulary of 100,000 words requires a co-occurrence matrix with 10 billion entries. Most of those entries are zero, because most pairs of words never appear near each other in any corpus. Sparse representations are expensive to store, slow to manipulate, and difficult to generalize from.

Skip-gram takes a different angle. Instead of storing counts, it trains a model to predict: given the word "fox," what words tend to appear nearby? The model is never asked to identify what "fox" means in the abstract. It is simply asked to predict context. But in learning to predict context well, it discovers which words share contextual patterns, and those shared patterns encode meaning.

The training setup is built around sliding windows. You take a sentence, pick a center word, and look at a fixed window around it. Every word in that window becomes a target the model should predict given the center word. This process is repeated for every word in the corpus, sweeping the window from left to right across each sentence.

The window size controls how far left and right from the center word we look for context. This choice has a deeper effect than it might appear, because it determines what kind of linguistic relationship the model is trained to capture.

Smaller windows (size 2) force the model to predict words that are grammatically related to the center word. Adjectives appear right before their nouns, objects appear right after their verbs, and determiners immediately precede the nouns they modify. By predicting these close neighbors, the model learns embeddings that reflect syntactic roles and relationships. Two words with similar syntactic functions, like "big" and "small" or "run" and "walk," will share similar close neighbors and therefore develop similar embeddings under a small window.

Larger windows (size 5 to 10) capture broader co-occurrences. Within a 10-word window, words that discuss the same topic but are not grammatically linked will appear together. In a paragraph about cooking, "recipe," "ingredients," "oven," and "temperature" may not be grammatical neighbors, but they are topical neighbors. Skip-gram with a large window learns embeddings that reflect topical or semantic similarity rather than syntactic similarity. The word "doctor" will be near "hospital" and "nurse" in semantic space, even though these words rarely appear as direct grammatical partners.

This trade-off has practical consequences for downstream tasks. Embeddings trained with small windows work better for syntactic tasks like part-of-speech tagging and parsing. Embeddings trained with large windows work better for semantic tasks like word similarity, analogy solving, and document classification. When you use a pretrained Word2Vec model, checking the window size used during training tells you something important about what kind of relationships the embeddings capture. The original Word2Vec paper used window sizes of 5 to 10 for most experiments, aiming for broad semantic coverage.

There is also a subtler consideration around word frequency. In any corpus, high-frequency words like "the," "a," and "is" appear constantly, and with a large window they become context words for almost everything. This can overwhelm the training signal. Mikolov et al. introduced subsampling to address this: frequent words are randomly discarded during training data generation with a probability that scales with their frequency. This prevents the most common words from dominating the gradient signal and allows rarer, more informative words to contribute more to the learning process.

One important implementation detail is that the sliding window should not cross sentence boundaries. If you naively apply a window of size 5 at the end of one sentence and the beginning of the next, you will generate spurious (center, context) pairs from words that are unrelated. Real implementations either split the corpus into sentences first or use special sentence-boundary tokens. In practice, the effect is modest for large corpora but matters when you want clean, interpretable embeddings.

The Skip-gram architecture is a shallow neural network with one hidden layer and no nonlinear activation function. Its apparent simplicity is misleading, because all the meaningful computation happens in the weight matrices, not in any deep stacking of transformations.

The network contains three components:

• Input layer: A one-hot vector of dimension $V$ (vocabulary size), with a 1 at the index of the center word and 0 everywhere else.
• Hidden layer: A weight matrix $\mathbf{W}$ of shape $V \times d$, where $d$ is the embedding dimension. Multiplying the one-hot input by $\mathbf{W}$ selects the $d$-dimensional row corresponding to the center word.
• Output layer: A second weight matrix $\mathbf{W}'$ of shape $d \times V$ that projects the hidden vector back to vocabulary size, producing a score for every word.

After training, we discard the output matrix and use $\mathbf{W}$ as our embedding lookup table. Each row of $\mathbf{W}$ is the embedding for the corresponding word.

The absence of a nonlinear activation function between the input embedding lookup and the output scoring is intentional. Adding nonlinearities would make the model harder to optimize and would not obviously improve the quality of the learned representations. The linear structure means the model is essentially learning a bilinear scoring function between center word embeddings and context word embeddings, and the geometry of that scoring function is what creates the useful structure in the embedding space.

One-hot encoding gives each word a unique, orthogonal representation. Word $w$ with vocabulary index $i$ is represented as a vector $\mathbf{x}$ of length $V$ where $x_i = 1$ and all other entries are 0. This is not a meaningful representation on its own; its purpose is purely to select a row from the embedding matrix.

Multiplying a one-hot vector $\mathbf{x}$ by the embedding matrix $\mathbf{W}$ is equivalent to selecting row $i$:

where:

• $\mathbf{h}$: the hidden layer vector of dimension $d$, which is the center word embedding
• $\mathbf{W}$: the input embedding matrix of shape $V \times d$
• $\mathbf{x}$: the one-hot input vector with a 1 at position $i$
• $\mathbf{v}_{w_I}$: the $i$-th row of $\mathbf{W}$, the embedding for input word $w_I$

This selection operation is the key to Skip-gram's efficiency. We never multiply a full one-hot vector through a large matrix in practice. We simply look up a row. Because the one-hot vector is all zeros except at one position, the matrix multiplication collapses to a single row retrieval. Modern implementations store word embeddings in a dedicated Embedding lookup table, which is exactly this: a parameterized table where each row is a learnable vector, and integer indexing is used to retrieve the relevant row.

The output matrix $\mathbf{W}'$ of shape $d \times V$ computes a score for each vocabulary word. For context word $w_O$ with vocabulary index $j$, the score is:

where:

• $u_j$: the unnormalized score (logit) for vocabulary word $j$ being a context word of $w_I$
• $\mathbf{v}'_{w_O}$: the $j$-th column of $\mathbf{W}'$, the context embedding of word $w_O$
• $\mathbf{h}$: the center word embedding from the hidden layer

So every word has two vector representations: an input embedding (a row of $\mathbf{W}$) used when the word is the center, and a context embedding (a column of $\mathbf{W}'$) used when it appears in someone else's context window. This dual representation is a fundamental design feature of Word2Vec. After training, you typically use the input embeddings ($\mathbf{W}$) as the final word vectors, though some practitioners average both matrices, or use the context embeddings for specific downstream tasks.

The use of separate input and context matrices deserves explanation, because it might seem redundant at first. Why should the embedding for "fox" as a center word differ from its embedding as a context word?

The reason is flexibility and symmetry breaking. If we used the same matrix for both roles, the model would be constrained to treat each word as both predictor and predicted in a symmetric way. With separate matrices, the model can learn that "fox" as a center word needs to be good at pointing toward animal-related words in context space, while "fox" as a context word needs to be pointed to by sentences that often contain fox-relevant contexts. These are related but not identical requirements, and separate matrices allow the model to satisfy them independently.

This also has an interesting geometric consequence. If you compute the dot product $\mathbf{v}_{w_I} \cdot \mathbf{v}'_{w_O}$, you are measuring how well word $w_I$ as center predicts word $w_O$ as context, which is not the same as asking whether $w_O$ as center predicts $w_I$ as context (that would be $\mathbf{v}_{w_O} \cdot \mathbf{v}'_{w_I}$). The model naturally handles asymmetric co-occurrence statistics this way.

To train the model, we need to convert the raw scores $u_j$ into a probability distribution over all vocabulary words. The softmax function accomplishes this by exponentiating all scores and normalizing:

where:

• $P(w_O | w_I)$: the probability of observing context word $w_O$ given center word $w_I$
• $u_{w_O} = \mathbf{v}'_{w_O} \cdot \mathbf{h}$: the dot-product score for context word $w_O$
• $\sum_{j=1}^{V} \exp(u_j)$: the normalizing constant (the partition function), summing exponential scores over all $V$ vocabulary words

The exponential function ensures all values are positive. Dividing by the sum ensures the values sum to 1, making the output a valid probability distribution. Words with higher dot-product scores with the center word receive higher probability mass. This is the mechanism by which training works: for a true (center, context) pair, we want the probability of the true context word to be high. We want the model to assign most probability mass to the word observed in context, which means the embedding for the true context word should have a high dot product with the center word embedding.

The softmax has a useful geometric interpretation: it measures alignment between the center word's embedding and each candidate context embedding. A context word whose embedding points in a similar direction to the center word embedding receives a high dot product and therefore high probability. As training progresses, the context embedding $\mathbf{v}'_{w_O}$ for a word that frequently appears near $w_I$ gets pushed to be more similar to $\mathbf{v}_{w_I}$. Words that never appear together receive gradients pushing their embeddings apart.

The denominator $\sum_{j=1}^{V} \exp(u_j)$ requires computing a score for every word in the vocabulary. For typical vocabularies of 100,000 to 1,000,000 words, this is extremely expensive. For each training example, you compute $V$ dot products between the center word embedding and every context embedding in $\mathbf{W}'$. For a corpus with billions of training pairs, this sum dominates the computation. A vocabulary of 1,000,000 words means 1,000,000 dot products per training step, totaling trillions of floating point operations across a full training run.

This bottleneck is not a quirk of a particular implementation choice; it is inherent to any exact softmax over a large vocabulary. The reason is that softmax is a global operation: you must query every word in order to normalize correctly. This is in contrast to classification problems with a small number of classes, where softmax is cheap.

This fundamental computational bottleneck is why the Negative Sampling and Hierarchical Softmax approximations were introduced alongside Skip-gram in the original Word2Vec paper. The theoretical objective described in this chapter is the correct formulation; in practice, you always use one of these approximations. The next chapters cover each approximation in detail, showing how they preserve the essential learning dynamics while making training tractable on realistic corpora.

Skip-gram maximizes the log-likelihood of observing the actual context words given each center word. For a training corpus of $T$ words, the objective is:

where:

• $T$: total number of words in the corpus
• $c$: the window size (context radius in each direction)
• $w_t$: the center word at position $t$
• $w_{t+j}$: a context word at offset $j$ from the center
• $\theta$: all model parameters (both embedding matrices $\mathbf{W}$ and $\mathbf{W}'$)

The outer sum iterates over every word position in the corpus. The inner sum iterates over every context offset within the window, generating up to $2c$ training signals per center word. For window size $c = 2$, this means up to four context words per center word (offsets $-2, -1, +1, +2$, where valid), giving up to four gradient updates per word occurrence.

The log transform converts products of probabilities into sums, which is both numerically more stable and mathematically cleaner. Maximizing this log-likelihood is equivalent to minimizing the negative log-likelihood, which is the loss function:

Substituting the softmax expression for $P(w_{t+j} | w_t)$:

where:

• $u_{w_{t+j}} = \mathbf{v}'_{w_{t+j}} \cdot \mathbf{v}_{w_t}$: the dot-product score for the true context word
• $\log \sum_{k=1}^{V} \exp(u_k)$: the log-partition function, which penalizes the model for spreading probability mass across many words rather than concentrating it on the true context word

The loss structure has a clean intuition. The first term $u_{w_{t+j}}$ increases when the true context word's score increases: the model is rewarded for making true pairs score high. The second term $\log \sum_k \exp(u_k)$ increases whenever any word scores high: the model is penalized for assigning high scores to words that are not the true context. Training pulls these in opposition, and the equilibrium is a model that assigns high scores to semantically related words.

To understand how Skip-gram learns, it helps to see the gradient of the loss with respect to the embeddings. For a single training pair $(w_I, w_O)$, the gradient with respect to the context embedding $\mathbf{v}'_{w_j}$ of vocabulary word $j$ is:

where:

• $\hat{y}_j = P(w_j | w_I)$: the predicted probability that word $j$ is the context word
• $\mathbf{1}[j = O]$: an indicator that equals 1 if $j$ is the true context word and 0 otherwise
• $\mathbf{h}$: the center word embedding

This gradient has an elegant interpretation. If the model assigns high probability to the true context word ($\hat{y}_O \approx 1$), the error signal is nearly zero and the embedding barely changes. If it assigns low probability ($\hat{y}_O \approx 0$), the error is large and the context embedding is pushed toward the center embedding. For all other words $j \neq O$, the gradient pushes their context embeddings away from the center embedding in proportion to how much probability they were incorrectly assigned.

The gradient with respect to the center word embedding $\mathbf{v}_{w_I} = \mathbf{h}$ accumulates contributions from every vocabulary word's context embedding, weighted by the prediction error:

This again requires iterating over all $V$ vocabulary words for each training pair, which is the source of the computational bottleneck. Negative Sampling replaces this full sum with a small sample, making the gradient computation tractable.

One of Skip-gram's most important strengths is that training data generation is entirely automatic. Any large text corpus becomes training data without manual labeling. The algorithm is:

1. Tokenize the corpus into words
2. Slide a window of size $2c + 1$ across each sentence
3. For each position $t$, pair the center word with each context word in the window
4. Each (center, context) pair becomes one training example

This self-supervised nature means you can train on Wikipedia, books, news articles, web crawls, or any other text. The training signal comes entirely from the statistical patterns in the text, with no human annotation needed. This property allows the method to scale: it means you can scale arbitrarily by adding more data, and more data consistently produces better embeddings.

The training data volume grows quickly. For a corpus of $T$ words with window size $c$, each word generates up to $2c$ training pairs, yielding up to $2cT$ pairs total. A Wikipedia dump with 2 billion tokens and window size 5 generates up to 20 billion training pairs. This scale is part of what makes Skip-gram work: the model sees each word from many different perspectives, building a rich picture of its contextual distribution.

The data generation strategy reflects a key assumption: the meaning of a word can be inferred from the company it keeps, a principle known as the distributional hypothesis. Words that appear near "hospital" include "doctor," "nurse," "patient," "medicine," and "surgery." If we train a model to predict these context words from "hospital," the model learns that the "hospital" vector must have something in common with all of them, encoding the semantic field of medicine.

Words sharing many context words will naturally develop similar embeddings, because they must predict similar distributions. "Doctor" and "physician" appear in nearly identical contexts, so their embeddings converge toward one another during training. "Bank" appears in both financial and riverine contexts, so its embedding ends up reflecting a blend of both usages, positioned between the financial vocabulary cluster and the geographical vocabulary cluster. The geometry of the embedding space directly encodes distributional structure. What we think of intuitively as "meaning" turns out, to a remarkable degree, to be capturable by these distributional patterns.

The model also implicitly learns syntactic structure. Adjectives appear before nouns, so all adjectives tend to appear in adjective-like contexts. Over time, adjective embeddings cluster together in the space, even if no two particular adjectives share frequent co-occurrences. The same happens for verbs, nouns, and prepositions. This latent syntactic structure in the embedding space is not explicitly trained for; it emerges from the distributional patterns in grammatical text.

In practice, the training data generation step includes one more important operation: subsampling of frequent words. High-frequency words like "the," "a," "in," and "of" appear so often that they end up as context words for almost every center word. This creates a lot of uninformative training pairs: knowing that "fox" appeared near "the" tells you almost nothing about the meaning of "fox," because "the" appears near everything.

Mikolov et al. introduced a subsampling heuristic that discards each word $w$ from the training data with probability:

where $f(w)$ is the frequency of word $w$ in the corpus and $t$ is a threshold (typically around $10^{-5}$). Words with frequencies much higher than $t$ are discarded with high probability, while rare words are almost never discarded. The effect is twofold: it speeds up training by generating fewer training pairs, and it improves embedding quality by giving rarer, more informative words a larger share of the gradient signal.

Let us build a complete Skip-gram implementation using PyTorch. We will train on the small corpus above and observe how embeddings develop.

First, set up the training tensors:

Even on this tiny corpus of six sentences, the model has learned that "cat" and "dog" are more similar to each other than either is to random words. The corpus explicitly connects them: "a dog and a cat can be friends" and "cats and dogs both make good pets" place them in near-identical contexts. On a real corpus with billions of words, these patterns accumulate into fine-grained semantic structure across the full vocabulary.

The key parameters for the Skip-gram model are:

• embed_dim: Dimensionality of the word vectors. Typical values are 50 to 300. Larger dimensions can capture more nuance but require more data and compute to estimate reliably. If you train on too little data with a high-dimensional embedding, many dimensions will remain random noise. Mikolov et al. used 300 dimensions for their best results on analogy tasks.
• window_size: Controls the context radius. Smaller windows (2 to 3) capture syntactic relationships; larger windows (5 to 10) capture semantic ones. The original Word2Vec paper used 5 for Skip-gram.
• learning_rate: Step size for gradient updates. Adam with learning rates around 0.001 to 0.05 works well. The original implementation used stochastic gradient descent with a learning rate that decays linearly from 0.025 to near zero over training.
• num_epochs / training_steps: More training generally improves quality, with diminishing returns beyond a point determined by corpus size. For large corpora, a single pass through the data is often sufficient.
• min_count: The minimum number of times a word must appear in the corpus to be included in the vocabulary. Rare words have unreliable embeddings because they appear too rarely to accumulate meaningful gradient signal. Typical values are 5 to 10.

Watching the loss curve as training progresses reveals how the model learns. The curve below shows the characteristic shape: rapid initial improvement as the model moves from random embeddings to something meaningful, followed by slower refinement as the model optimizes finer details.

One of the most striking properties of Skip-gram embeddings, discovered empirically by Mikolov et al., is that meaningful structure emerges in the geometry of the learned space. This goes beyond individual similarities between word pairs. The relationships between words can be computed with vector arithmetic.

The most famous example is:

The difference vector $\vec{\text{king}} - \vec{\text{man}}$ captures the concept of royalty without gender. Adding $\vec{\text{woman}}$ moves from the "generic person" direction to the "woman" direction. The result lands near "queen" in the embedding space.

This works because of how the embeddings are learned. "King" appears in contexts involving royalty, power, and masculine roles. "Queen" appears in contexts involving royalty, power, and feminine roles. The difference between their embeddings captures the gender axis, not because the model was trained on gender or royalty explicitly, but because these distributional differences emerged from the corpus statistics.

Vector analogies generalize beyond gender. They work for country-capital relationships (France: Paris :: Germany: Berlin), verb tenses (run: ran :: swim: swam), and singular-plural forms (cat: cats :: dog: dogs). The model has implicitly learned grammatical and factual structure, encoded as directions in the embedding space.

It is important to temper this with realistic expectations. The analogy task works impressively for common, high-frequency relationships, but it fails in many edge cases. Less frequent words have noisier embeddings. Polysemous words occupy averaged positions in the space that do not fully capture any single sense. And the analogy test evaluates a narrow property of the embeddings; word similarity benchmarks and downstream task performance are more reliable indicators of embedding quality.

Another way to understand the embedding space is to look at nearest neighbors for specific words. In well-trained Skip-gram embeddings on a large corpus, the nearest neighbors of "Paris" include "London," "Berlin," "Rome," and other European capitals. The nearest neighbors of "running" include "jogging," "sprinting," and "walking." The nearest neighbors of "terrible" include "awful," "horrible," and "dreadful."

These neighborhoods capture different types of relationships depending on window size. Under small windows, "running" might be near "quickly" (because adverbs follow verb phrases) as well as other verbs. Under large windows, "running" might be near "marathon," "race," and "athlete" (because these appear in the same topical contexts).

This is why pretrained embeddings carry implicit choices about the linguistic relationships they capture. When you download a Word2Vec model trained on Google News with a particular configuration, you are implicitly accepting the trade-offs made in window size, dimensionality, and subsampling threshold. Understanding these trade-offs helps you decide whether a given pretrained model is appropriate for your downstream task.

Word2Vec comes in two flavors: Skip-gram (center predicts context) and CBOW (Continuous Bag of Words: context predicts center). Understanding the intuitive difference helps you choose the right one for your task.

Skip-gram treats each context word prediction as an independent task. For a center word with four context words in the window, Skip-gram makes four separate predictions, each generating its own gradient. This creates more training examples per center word occurrence, which directly benefits rare words. Even if a rare word appears only a handful of times in the corpus, each occurrence contributes multiple gradient updates, pulling the embedding toward a more informative position in the space.

CBOW averages all context word embeddings together before making a single prediction of the center word. This averaging smooths over individual context words and makes the model less sensitive to any single co-occurrence. The model processes the full context in one forward pass, making it computationally faster per training step. However, the averaging also dilutes learning signal for rare words, because their contributions get mixed with those of many more common words.

The averaging operation in CBOW also means that word order within the window is lost. Whether "brown fox" or "fox brown" appears in the context does not matter; both produce the same averaged context vector. Skip-gram preserves this positional information implicitly because it generates separate training pairs for each context position. In practice this distinction is minor for most downstream tasks, but it matters in linguistic analyses that use the embeddings to study word order effects.

The practical implications for choosing between the two are:

• Skip-gram learns better representations for infrequent words and morphologically complex words. Prefer it when your downstream task involves rare terminology or you have limited training data.
• CBOW trains faster and often produces competitive results for high-frequency words. Prefer it when training speed matters and your vocabulary is dominated by common words.
• For most general-purpose embedding tasks on large corpora, the two perform comparably. Skip-gram's advantage is most pronounced for vocabulary at the rare end of the frequency distribution.

The next chapter covers the CBOW architecture in detail, including its forward pass, gradient derivation, and implementation.

To make the mathematics concrete, let us trace through a single gradient update for a specific training pair. Suppose our center word is "cat" (index 0) and the true context word is "dog" (index 3), and we have a toy vocabulary of four words: ["cat", "mat", "mouse", "dog"].

We start with random embeddings. Say the center embedding for "cat" is $\mathbf{h} = [0.1, -0.2, 0.3]$ and the context embeddings for all four words are:

• $\mathbf{v}'_{\text{cat}} = [0.0, 0.1, -0.1]$
• $\mathbf{v}'_{\text{mat}} = [-0.1, 0.0, 0.2]$
• $\mathbf{v}'_{\text{mouse}} = [0.2, -0.1, 0.0]$
• $\mathbf{v}'_{\text{dog}} = [-0.2, 0.3, 0.1]$

Step 1: Compute raw scores by taking dot products:

Step 2: Apply softmax to get probabilities:

The model assigns roughly equal probability to all four words, which makes sense since the embeddings are random.

Step 3: Compute the prediction error. The true context word is "dog" (index 3), so:

Step 4: Update context embeddings. For "dog" (the true context word), the negative error ($-0.762$) means we push its context embedding toward the center embedding. For all other words, positive errors mean we push their context embeddings slightly away from the center embedding.

Step 5: Update the center embedding for "cat" by accumulating the error-weighted context embeddings across all vocabulary words. This is the expensive step that requires iterating over all $V$ words.

After this one update, the context embedding for "dog" has moved slightly closer to the center embedding of "cat." After millions of such updates across the full corpus, the alignment between frequently co-occurring words becomes strong, while rarely or never co-occurring words become repelled in the embedding space.

Skip-gram's full softmax objective is computationally intractable for realistic vocabularies. Computing $\sum_{j=1}^{V} \exp(u_j)$ over a vocabulary of 1,000,000 words, for every training pair, across billions of training examples, would require months of computation. This is why the Negative Sampling and Hierarchical Softmax approximations were introduced alongside Skip-gram in the original paper. The theoretical objective described in this chapter is the ideal form; in practice, you always use one of these approximations.

Beyond computational cost, Skip-gram embeddings are context-insensitive. Each word receives a single vector regardless of how it is used in a given sentence. The word "bank" gets one embedding that must simultaneously encode financial institutions and river banks. In a sentence about money, the model cannot distinguish "bank" as a financial entity from "bank" used in a geographic context. This is a fundamental limitation of the representation, not a training issue. No matter how much data you train on, a single-vector representation cannot fully capture a polysemous word.

Morphologically rich languages present another challenge. In languages like German, Turkish, or Finnish, a single root word can generate dozens of inflected forms. Each form receives its own Skip-gram embedding with no parameter sharing between forms of the same root. "Run," "runs," "ran," "running," and "runner" are five separate entries in the vocabulary, each with independently learned embeddings. While the corpus statistics will push these embeddings to be similar (since they appear in overlapping contexts), they do not share any parameters, so each requires enough data to be well-trained independently. Subword-based approaches like FastText, which represents words as sums of character n-gram embeddings, address this by sharing parameters across morphological variants.

Skip-gram also assumes context words within the window are independent given the center word. The model treats "predicting 'quick' given 'fox'" and "predicting 'brown' given 'fox'" as completely separate tasks. In reality, "quick" and "brown" co-occur together as part of the phrase "the quick brown fox," and that co-occurrence pattern carries information beyond what Skip-gram captures. More sophisticated models that condition jointly on the full context address this limitation, at the cost of much greater complexity.

Despite these limitations, Skip-gram changed the field. Before Word2Vec, word representations were typically high-dimensional, sparse, and brittle. Skip-gram demonstrated that a two-layer neural network, trained on a self-supervised prediction task with no human labels, could produce dense vectors capturing subtle semantic relationships at scale. The representations transferred across tasks: embeddings trained on Wikipedia improved sentiment analysis, named entity recognition, machine translation, question answering, and dozens of other applications without task-specific supervision. This transfer learning paradigm, enabled by Skip-gram, became the blueprint for modern NLP.

The scale at which Skip-gram could be trained was also unprecedented. Mikolov et al. trained on a Google News corpus of 100 billion words. The resulting vocabulary contained 3 million words, each represented as a 300-dimensional vector. No previous word representation method had operated at this scale or produced results of comparable quality. The combination of the efficient training algorithm, the self-supervised objective, and the scalable architecture created a step change in what word representations could do.

Skip-gram also established a design pattern that influenced everything that followed. The key insight was that self-supervised prediction tasks, where the training signal comes from the structure of the data itself, could teach a model about meaning. This insight resurfaced in BERT (predicting masked words), GPT (predicting the next word), and the broader paradigm of pretraining on prediction tasks followed by fine-tuning on specific applications that defines modern NLP. Every large language model you interact with today is, in a conceptual sense, a descendant of Skip-gram: a model trained to predict part of the text from the rest, in a process that incidentally teaches it rich representations of language.

The Word2Vec codebase released by Google in 2013 combined an academic contribution with a practical tool that NLP practitioners could download and use immediately. Within months of release, it transformed how practitioners approached NLP tasks. Pre-training on large corpora and fine-tuning on small labeled datasets became standard practice. The concept of "semantic space" moved from a theoretical curiosity to an engineering tool. The embeddings could be visualized with t-SNE, interrogated with analogy queries, and used as features in virtually any downstream model. This combination of theoretical elegance, practical utility, and strong empirical results made Word2Vec one of the most cited and influential papers in the history of NLP.

Skip-gram learns word embeddings by training a shallow neural network to predict context words from a center word. The key ideas are:

• Training signal: Every word occurrence generates multiple (center, context) training pairs via a sliding window. No manual labels are required; the corpus provides its own supervision through the distributional structure of language.
• Architecture: Two embedding matrices (center embeddings $\mathbf{W}$ and context embeddings $\mathbf{W}'$) are learned jointly. After training, the center embeddings serve as the word representations, with each row corresponding to one word's $d$-dimensional vector.
• Objective: Maximize the log-likelihood of observed (center, context) pairs using softmax over the full vocabulary. This pulls together embeddings of words that co-occur and pushes apart embeddings of words that do not.
• Window size: A critical hyperparameter controlling whether the model captures syntactic (small window) or semantic (large window) relationships. The choice shapes the kind of similarity the embeddings encode.
• Dual representations: Every word has a center embedding (used when predicting context) and a context embedding (used when being predicted). This asymmetry allows the model to capture directional co-occurrence patterns.
• Skip-gram vs. CBOW: Skip-gram makes one prediction per context position, generating more learning signals per occurrence and benefiting rare words. CBOW averages context embeddings and makes one prediction per center word, training faster but with less benefit for rare vocabulary.
• Subsampling: Discarding frequent words during training data generation reduces noise and gives rarer, more informative words greater influence on the embeddings.

The full softmax objective described here is exact but computationally expensive. The next chapters cover Negative Sampling and Hierarchical Softmax, which make Skip-gram tractable at the scale of real-world corpora while preserving the essential learning dynamics.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about the Skip-gram model.