Word Embeddings: Word2Vec, GloVe and Vector Semantics › Word2Vec: Learning Embeddings from Local Context

Word2Vec, introduced by Mikolov et al. in 2013, learns word embeddings by predicting words from their local context. The key insight is simple: train a neural network to predict surrounding words, and the learned weights become meaningful word representations.

Word2Vec offers two architectures: Continuous Bag of Words (CBOW) and Skip-gram. Both use shallow neural networks, but they solve inverse problems:

• CBOW: Predicts the center word from surrounding context words
• Skip-gram: Predicts surrounding context words from the center word

Skip-gram typically performs better on rare words and is more commonly used, so we'll focus on it. The principles apply to both.

Let's start with a simple observation. When you read the sentence "The quick brown fox jumps over the lazy dog," you immediately understand that "fox" is related to "quick," "brown," "jumps," and "over" because they appear together. This is the distributional hypothesis in action: words that appear in similar contexts tend to have similar meanings.

Word2Vec formalizes this intuition through a prediction task. Instead of asking "what do these words mean?", we ask "can we predict which words appear together?" The key insight is that learning to predict context automatically discovers semantic relationships. Words that appear in similar contexts will need similar internal representations to make accurate predictions, and these representations become our embeddings.

Skip-gram takes a center word and tries to predict words that appear nearby in the text. Consider our example sentence: "The quick brown fox jumps over the lazy dog." If "fox" is our center word and we use a window size of 2, we want to predict: "quick", "brown", "jumps", "over".

The architecture is surprisingly simple, consisting of just three layers:

1. Input layer: One-hot encoded center word (vocabulary size $V$). This is a sparse vector where exactly one position is 1, indicating which word we're currently considering.

2. Hidden layer: A linear projection to embedding dimension $d$ (typically 100-300). When we multiply the one-hot vector by the weight matrix, we're selecting one row from that matrix, which is the embedding vector for our center word.

3. Output layer: A softmax over the entire vocabulary to predict which word appears in the context. This gives us a probability distribution over all possible context words.

The key insight is that the weight matrix connecting input to hidden layer becomes our word embeddings. Each row of this matrix is the embedding vector for one word. As the network learns to predict context words accurately, it must learn to place similar words (those with similar contexts) in similar regions of embedding space.

Now let's formalize this intuition into a mathematical objective. We want to maximize the probability of observing the actual context words given our center word. For a center word $w_t$ at position $t$ and a context window of size $c$, we observe context words $w_{t-c}, \ldots, w_{t-1}, w_{t+1}, \ldots, w_{t+c}$.

Our goal is to maximize:

$$P(w_{t-c}, \ldots, w_{t-1}, w_{t+1}, \ldots, w_{t+c} | w_t)$$

This probability tells us: "Given that we see word $w_t$ as the center word, how likely are we to see these specific context words around it?" To make this tractable, we make a simplifying assumption: context words are independent given the center word. This means the probability of seeing "quick" and "brown" around "fox" is the product of seeing each individually:

$$P(w_{t-c}, \ldots, w_{t-1}, w_{t+1}, \ldots, w_{t+c} | w_t) = \prod_{j=-c, j \neq 0}^{c} P(w_{t+j} | w_t)$$

where:

• $w_t$: the center word at position $t$ in the corpus
• $w_{t+j}$: a context word at position $t+j$ (within the window around $w_t$)
• $c$: the context window size (typically 5-10 words on each side)
• $P(w_{t+j} | w_t)$: the conditional probability of observing context word $w_{t+j}$ given center word $w_t$

This assumption isn't perfectly true (words in context aren't truly independent), but it works well in practice and makes the math manageable. Each term $P(w_{t+j} | w_t)$ asks: "Given center word $w_t$, what's the probability that word $w_{t+j}$ appears in the context?"

To convert this into a loss function we can minimize (standard practice in machine learning), we take the negative logarithm:

$$L = -\sum_{j=-c, j \neq 0}^{c} \log P(w_{t+j} | w_t)$$

where:

• $L$: the loss function to minimize
• The negative sign converts maximization (of probabilities) to minimization (standard in optimization)
• The logarithm converts the product into a sum, which is numerically more stable and has better gradient properties

Why the negative logarithm? Two reasons: (1) maximizing probabilities is equivalent to minimizing negative log probabilities, and (2) logarithms convert products into sums, which are easier to work with numerically and have better gradient properties.

Now we need to compute $P(w_{t+j} | w_t)$, the probability that a specific context word appears given the center word. This is where embeddings enter the picture.

We compute this probability using the softmax function over the entire vocabulary:

$$P(w_O | w_I) = \frac{\exp(\mathbf{v}_{w_O}^T \mathbf{v}_{w_I})}{\sum_{w=1}^{V} \exp(\mathbf{v}_w^T \mathbf{v}_{w_I})}$$

where:

• $w_I$: the input (center) word
• $w_O$: the output (context) word
• $\mathbf{v}_{w_I} \in \mathbb{R}^d$: the input embedding vector for word $w_I$ (center word representation), where $d$ is the embedding dimension
• $\mathbf{v}_{w_O} \in \mathbb{R}^d$: the output embedding vector for word $w_O$ (context word representation)
• $\mathbf{v}_{w_O}^T \mathbf{v}_{w_I}$: the dot product (scalar) measuring similarity between the two embedding vectors
• $V$: the vocabulary size (total number of unique words)
• $\exp(\cdot)$: the exponential function, which amplifies differences in dot products
• The numerator $\exp(\mathbf{v}_{w_O}^T \mathbf{v}_{w_I})$ measures the "compatibility" between center and context words
• The denominator $\sum_{w=1}^{V} \exp(\mathbf{v}_w^T \mathbf{v}_{w_I})$ normalizes over all possible context words so the probabilities sum to 1

This is the softmax function, which converts raw similarity scores (dot products) into a probability distribution over the vocabulary.

Why the dot product? The dot product $\mathbf{v}_{w_O}^T \mathbf{v}_{w_I}$ measures how similar two vectors are. If embeddings are similar (point in similar directions), the dot product is large. If they're different (point in different directions), the dot product is small. The exponential function amplifies these differences, making similar word pairs have much higher probabilities.

Why two embedding matrices? Notice that Word2Vec actually learns two embedding matrices: one for input words (center words) and one for output words (context words). This asymmetry allows the model to learn different representations for the same word depending on whether it's being used as a center word or a context word. In practice, we typically use the input matrix as our final word embeddings, though some applications average or concatenate both.

The softmax formulation has a significant problem: it's computationally expensive. To compute $P(w_O | w_I)$, we need to evaluate the dot product between the center word embedding and every single word in the vocabulary, then exponentiate each result. With a vocabulary of $V = 50,000$ words, this means 50,000 dot products and 50,000 exponentiations for every single training example. For a corpus with millions of training examples, this becomes prohibitively slow.

We need a way to approximate the softmax that:

1. Is computationally efficient (doesn't require evaluating all vocabulary words)
2. Still learns meaningful embeddings
3. Captures the same semantic relationships

Negative sampling solves this by reframing the problem. Instead of asking "what's the probability of each word appearing in context?" (a multi-class classification problem), we ask "is this word likely to appear in context, or not?" (a binary classification problem).

Here's how it works:

1. Positive examples: The actual context words we observe are treated as positive examples. We want the model to assign high probability to these.

2. Negative examples: We randomly sample $k$ words from the vocabulary (typically $k = 5-20$) that did NOT appear in the context. These are negative examples, words we want the model to assign low probability to.

3. Binary classification: For each positive and negative example, we train a binary classifier using the sigmoid function instead of softmax.

The loss function becomes:

$$L = -\log \sigma(\mathbf{v}_{w_O}^T \mathbf{v}_{w_I}) - \sum_{i=1}^{k} \log \sigma(-\mathbf{v}_{w_i}^T \mathbf{v}_{w_I})$$

where:

• $L$: the negative sampling loss function
• $\sigma(x) = \frac{1}{1 + e^{-x}}$: the sigmoid function, which maps any real number to the interval $(0, 1)$
• $w_I$: the center (input) word
• $w_O$: the positive context word (observed in the actual context)
• $\mathbf{v}_{w_I} \in \mathbb{R}^d$: the input embedding vector for the center word
• $\mathbf{v}_{w_O} \in \mathbb{R}^d$: the output embedding vector for the positive context word
• $w_i$: the $i$-th negative sample word (randomly sampled from vocabulary, not in context)
• $\mathbf{v}_{w_i} \in \mathbb{R}^d$: the output embedding vector for negative sample $i$
• $k$: the number of negative samples (typically 5-20)

Let's understand each term:

• First term $-\log \sigma(\mathbf{v}_{w_O}^T \mathbf{v}_{w_I})$: We want to maximize the probability that the positive context word $w_O$ appears with center word $w_I$. When the dot product $\mathbf{v}_{w_O}^T \mathbf{v}_{w_I}$ is large (positive), $\sigma(\cdot)$ approaches 1, so $-\log \sigma(\cdot)$ approaches 0 (minimizing the loss, which is good).

• Second term $-\sum_{i=1}^{k} \log \sigma(-\mathbf{v}_{w_i}^T \mathbf{v}_{w_I})$: We want to minimize the probability that negative words $w_i$ appear with center word $w_I$. The negative sign before the dot product in the sigmoid argument is key: we compute $\sigma(-\mathbf{v}_{w_i}^T \mathbf{v}_{w_I})$ instead of $\sigma(\mathbf{v}_{w_i}^T \mathbf{v}_{w_I})$. When the dot product $\mathbf{v}_{w_i}^T \mathbf{v}_{w_I}$ is large (positive), $-\mathbf{v}_{w_i}^T \mathbf{v}_{w_I}$ is large and negative, so $\sigma(-\mathbf{v}_{w_i}^T \mathbf{v}_{w_I})$ approaches 0, making $-\log \sigma(\cdot)$ large. Since we're minimizing the loss, this encourages the dot product between center word and negative samples to be small (or negative), pushing their embeddings apart in vector space.

Why does this work? The key insight is that most words are unrelated to any given center word. By explicitly teaching the model that random words are negative examples, we implicitly learn that words with high dot products (similar embeddings) are likely to be related. The model learns to pull related words together in embedding space and push unrelated words apart.

Computational savings: Instead of $O(V)$ operations per training example, we now need only $O(k+1)$ operations. With $V = 50,000$ and $k = 5$, this is approximately an 8,000x speedup (50,000 / 6 ≈ 8,333), making training feasible on large corpora.

Word2Vec training proceeds as follows:

1. Initialize embeddings: Random vectors for each word in vocabulary
2. Slide window: Move a context window across the training corpus
3. For each center word:
   • Sample positive context words from the window
   • Sample negative words from the vocabulary (weighted by frequency)
   • Update embeddings using gradient descent
4. Repeat until convergence

The key hyperparameters are:

• Embedding dimension ($d$): Typically 100-300. Larger dimensions capture more nuance but require more data
• Window size ($c$): Typically 5-10. Larger windows capture more global relationships
• Negative samples ($k$): Typically 5-20. More negatives improve quality but slow training
• Learning rate: Typically 0.01-0.05, often with decay