Skip-gram Model › Worked Example: Tracing a Single Gradient Update

To make the mathematics concrete, let us trace through a single gradient update for a specific training pair. Suppose our center word is "cat" (index 0) and the true context word is "dog" (index 3), and we have a toy vocabulary of four words: ["cat", "mat", "mouse", "dog"].

We start with random embeddings. Say the center embedding for "cat" is $\mathbf{h} = [0.1, -0.2, 0.3]$ and the context embeddings for all four words are:

• $\mathbf{v}'_{\text{cat}} = [0.0, 0.1, -0.1]$
• $\mathbf{v}'_{\text{mat}} = [-0.1, 0.0, 0.2]$
• $\mathbf{v}'_{\text{mouse}} = [0.2, -0.1, 0.0]$
• $\mathbf{v}'_{\text{dog}} = [-0.2, 0.3, 0.1]$

Step 1: Compute raw scores by taking dot products:

$$\begin{aligned} u_{\text{cat}} &= \mathbf{h} \cdot \mathbf{v}'_{\text{cat}} = (0.1)(0.0) + (-0.2)(0.1) + (0.3)(-0.1) = -0.05 \\ u_{\text{mat}} &= \mathbf{h} \cdot \mathbf{v}'_{\text{mat}} = (0.1)(-0.1) + (-0.2)(0.0) + (0.3)(0.2) = 0.05 \\ u_{\text{mouse}} &= \mathbf{h} \cdot \mathbf{v}'_{\text{mouse}} = (0.1)(0.2) + (-0.2)(-0.1) + (0.3)(0.0) = 0.04 \\ u_{\text{dog}} &= \mathbf{h} \cdot \mathbf{v}'_{\text{dog}} = (0.1)(-0.2) + (-0.2)(0.3) + (0.3)(0.1) = -0.05 \end{aligned}$$

Step 2: Apply softmax to get probabilities:

$$\begin{aligned} P &= \text{softmax}([-0.05, 0.05, 0.04, -0.05]) \\ &\approx [0.238, 0.263, 0.260, 0.238] \end{aligned}$$

The model assigns roughly equal probability to all four words, which makes sense since the embeddings are random.

Step 3: Compute the prediction error. The true context word is "dog" (index 3), so:

$$e_j = \hat{y}_j - \mathbf{1}[j = \text{dog}] = [0.238 - 0, \; 0.263 - 0, \; 0.260 - 0, \; 0.238 - 1] = [0.238, 0.263, 0.260, -0.762]$$

Step 4: Update context embeddings. For "dog" (the true context word), the negative error ($-0.762$) means we push its context embedding toward the center embedding. For all other words, positive errors mean we push their context embeddings slightly away from the center embedding.

Step 5: Update the center embedding for "cat" by accumulating the error-weighted context embeddings across all vocabulary words. This is the expensive step that requires iterating over all $V$ words.

After this one update, the context embedding for "dog" has moved slightly closer to the center embedding of "cat." After millions of such updates across the full corpus, the alignment between frequently co-occurring words becomes strong, while rarely or never co-occurring words become repelled in the embedding space.