Skip-gram Model › The Objective Function

Skip-gram maximizes the log-likelihood of observing the actual context words given each center word. For a training corpus of $T$ words, the objective is:

$$J(\theta) = \frac{1}{T} \sum_{t=1}^{T} \sum_{\substack{-c \leq j \leq c \\ j \neq 0}} \log P(w_{t+j} | w_t;\, \theta)$$

where:

• $T$: total number of words in the corpus
• $c$: the window size (context radius in each direction)
• $w_t$: the center word at position $t$
• $w_{t+j}$: a context word at offset $j$ from the center
• $\theta$: all model parameters (both embedding matrices $\mathbf{W}$ and $\mathbf{W}'$)

The outer sum iterates over every word position in the corpus. The inner sum iterates over every context offset within the window, generating up to $2c$ training signals per center word. For window size $c = 2$, this means up to four context words per center word (offsets $-2, -1, +1, +2$, where valid), giving up to four gradient updates per word occurrence.

The log transform converts products of probabilities into sums, which is both numerically more stable and mathematically cleaner. Maximizing this log-likelihood is equivalent to minimizing the negative log-likelihood, which is the loss function:

$$\mathcal{L}(\theta) = -\frac{1}{T} \sum_{t=1}^{T} \sum_{\substack{-c \leq j \leq c \\ j \neq 0}} \log P(w_{t+j} | w_t;\, \theta)$$

Substituting the softmax expression for $P(w_{t+j} | w_t)$:

$$\mathcal{L}(\theta) = -\frac{1}{T} \sum_{t=1}^{T} \sum_{\substack{-c \leq j \leq c \\ j \neq 0}} \Bigl[ u_{w_{t+j}} - \log \sum_{k=1}^{V} \exp(u_k) \Bigr]$$

where:

• $u_{w_{t+j}} = \mathbf{v}'_{w_{t+j}} \cdot \mathbf{v}_{w_t}$: the dot-product score for the true context word
• $\log \sum_{k=1}^{V} \exp(u_k)$: the log-partition function, which penalizes the model for spreading probability mass across many words rather than concentrating it on the true context word

The loss structure has a clean intuition. The first term $u_{w_{t+j}}$ increases when the true context word's score increases: the model is rewarded for making true pairs score high. The second term $\log \sum_k \exp(u_k)$ increases whenever any word scores high: the model is penalized for assigning high scores to words that are not the true context. Training pulls these in opposition, and the equilibrium is a model that assigns high scores to semantically related words.

To understand how Skip-gram learns, it helps to see the gradient of the loss with respect to the embeddings. For a single training pair $(w_I, w_O)$, the gradient with respect to the context embedding $\mathbf{v}'_{w_j}$ of vocabulary word $j$ is:

$$\frac{\partial \mathcal{L}}{\partial \mathbf{v}'_{w_j}} = (\hat{y}_j - \mathbf{1}[j = O]) \cdot \mathbf{h}$$

where:

• $\hat{y}_j = P(w_j | w_I)$: the predicted probability that word $j$ is the context word
• $\mathbf{1}[j = O]$: an indicator that equals 1 if $j$ is the true context word and 0 otherwise
• $\mathbf{h}$: the center word embedding

This gradient has an elegant interpretation. If the model assigns high probability to the true context word ($\hat{y}_O \approx 1$), the error signal is nearly zero and the embedding barely changes. If it assigns low probability ($\hat{y}_O \approx 0$), the error is large and the context embedding is pushed toward the center embedding. For all other words $j \neq O$, the gradient pushes their context embeddings away from the center embedding in proportion to how much probability they were incorrectly assigned.

The gradient with respect to the center word embedding $\mathbf{v}_{w_I} = \mathbf{h}$ accumulates contributions from every vocabulary word's context embedding, weighted by the prediction error:

$$\frac{\partial \mathcal{L}}{\partial \mathbf{h}} = \sum_{j=1}^{V} (\hat{y}_j - \mathbf{1}[j = O]) \cdot \mathbf{v}'_{w_j}$$

This again requires iterating over all $V$ vocabulary words for each training pair, which is the source of the computational bottleneck. Negative Sampling replaces this full sum with a small sample, making the gradient computation tractable.