GloVe › Deriving the GloVe Objective

The derivation starts from the ratios and works backward to a tractable optimization objective. Each step follows necessarily from a design choice, so the final formula is not arbitrary: it is the natural consequence of the ratio insight combined with a small set of reasonable constraints.

We want word vectors $\vec{w}_i$, $\vec{w}_j$, and $\vec{w}_k$ such that some function $F$ of these vectors recovers the ratio:

$$F(\vec{w}_i, \vec{w}_j, \vec{w}_k) = \frac{P_{ik}}{P_{jk}}$$

where:

• $\vec{w}_i$: the word vector for word $i$ (the first target word)
• $\vec{w}_j$: the word vector for word $j$ (the second target word we are comparing against)
• $\vec{w}_k$: the vector for the probe word $k$
• $P_{ik} = P(k \mid i) = X_{ik}/X_i$: the conditional probability of seeing $k$ in the context of $i$

We want this function to depend on the difference $\vec{w}_i - \vec{w}_j$, because we are comparing two target words and the comparison should be captured as a vector difference. The simplest way to combine a difference vector with a third vector is via a dot product:

$$F\left((\vec{w}_i - \vec{w}_j)^T \tilde{\vec{w}}_k\right) = \frac{P_{ik}}{P_{jk}}$$

Here $\tilde{\vec{w}}_k$ is a separate vector for word $k$ when it is a context word, distinct from $\vec{w}_k$ when $k$ is the target word. This distinction is standard in embedding models: each word plays two roles (as a target and as a context), and the two roles are learned separately. Using separate vectors for the two roles avoids conflating them and is a key part of why the derivation works out cleanly.

We need $F$ to satisfy the ratio decomposition property. When we expand the ratio $P_{ik}/P_{jk}$, we are taking the ratio of two quantities that each involve a single target word and the probe word $k$. So we need:

$$F\left((\vec{w}_i - \vec{w}_j)^T \tilde{\vec{w}}_k\right) = \frac{F(\vec{w}_i^T \tilde{\vec{w}}_k)}{F(\vec{w}_j^T \tilde{\vec{w}}_k)}$$

This is a functional equation: we need a function $F$ such that $F(a - b) = F(a)/F(b)$ for all $a$ and $b$. The exponential function satisfies this exactly, since:

$$\exp(a - b) = \frac{\exp(a)}{\exp(b)}$$

This is the only continuous function (up to a constant multiplier) that converts subtraction into division. The uniqueness here is reassuring: we are not choosing $\exp$ arbitrarily, it is forced by the design requirement.

Plugging in $F = \exp$ and setting the result equal to the true ratio $P_{ik}/P_{jk}$ gives us, for each individual word pair $(i, k)$:

$$\exp(\vec{w}_i^T \tilde{\vec{w}}_k) = P_{ik} = \frac{X_{ik}}{X_i}$$

Taking the natural logarithm of both sides:

$$\begin{aligned} \vec{w}_i^T \tilde{\vec{w}}_k &= \log P_{ik} \\ &= \log \frac{X_{ik}}{X_i} \\ &= \log X_{ik} - \log X_i \end{aligned}$$

The term $\log X_i$ depends only on word $i$, not on the context word $k$. It represents word $i$'s overall frequency as a context center: a word that appears in many contexts has a large $X_i$. We can absorb this word-frequency effect into a scalar bias $b_i$ learned during training. Similarly, by symmetry, we add a context bias $\tilde{b}_k$ for word $k$'s frequency as a context word:

$$\vec{w}_i^T \tilde{\vec{w}}_k + b_i + \tilde{b}_k = \log X_{ik}$$

where:

• $b_i$: a scalar bias for word $i$ that absorbs its context frequency $\log X_i$
• $\tilde{b}_k$: a scalar bias for word $k$ in its role as a context word

The context bias $\tilde{b}_k$ is not strictly required by the derivation, but it restores symmetry. Because $X_{ik} = X_{ki}$ for symmetric windows, both the target role and the context role of each word should have their own frequency-absorbing bias. Adding $\tilde{b}_k$ makes the formulation symmetric in the roles of target and context.

This equation is the core relationship GloVe trains for. We want, for every word pair $(i, j)$, the model's prediction $\vec{w}_i^T \tilde{\vec{w}}_j + b_i + \tilde{b}_j$ to equal the corpus-derived target $\log X_{ij}$.

We want our word vectors and biases to satisfy $\vec{w}_i^T \tilde{\vec{w}}_j + b_i + \tilde{b}_j \approx \log X_{ij}$ for all $(i, j)$ pairs. The natural formulation is a least squares objective over all $V^2$ word pairs:

$$J = \sum_{i,j=1}^{V} \left(\vec{w}_i^T \tilde{\vec{w}}_j + b_i + \tilde{b}_j - \log X_{ij}\right)^2$$

But this treats all co-occurrence pairs equally, and that creates two serious problems. First, rare co-occurrences (small $X_{ij}$) are statistically noisy. If "quasar" and "kumquat" appeared together once in a 100-billion-token corpus, that single co-occurrence is almost certainly an accident, not evidence of a semantic relationship. Yet the unweighted objective would treat this pair as equally important as "cat" co-occurring with "dog" 10,000 times. Second, most entries of $X$ are zero (words that never co-occurred), and $\log 0$ is undefined, so the plain objective is not even computable.

The solution is to weight each term by a function $f(X_{ij})$ that satisfies four properties:

1. Zero co-occurrences get zero weight, so $\log 0$ never appears in the sum
2. Rare co-occurrences get lower weight, reducing the influence of statistical noise
3. Very frequent co-occurrences do not get excessive weight (stop words like "the" co-occur with almost everything, but without a semantic relationship)
4. The function is continuous and non-decreasing

This gives the final GloVe objective:

$$J = \sum_{i,j=1}^{V} f(X_{ij}) \left(\vec{w}_i^T \tilde{\vec{w}}_j + b_i + \tilde{b}_j - \log X_{ij}\right)^2$$

where:

• $f(X_{ij})$: the weighting function that scales each term's contribution by the reliability of the co-occurrence count
• $\vec{w}_i^T \tilde{\vec{w}}_j + b_i + \tilde{b}_j$: the model's prediction of $\log X_{ij}$
• $\log X_{ij}$: the target value derived from the corpus

This is the weighted least squares objective that GloVe minimizes. Because $f(0) = 0$, the sum effectively runs only over pairs where $X_{ij} > 0$. For large vocabularies this is a small fraction of all $V^2$ pairs, making the sum tractable.