GloVe: Global Vectors for Word Representation

Word2Vec learns embeddings through local context windows, predicting surrounding words one pair at a time. This approach works well but ignores a fundamental insight: some word relationships are global properties of the corpus. The word "ice" might appear near "cold" millions of times across a billion-word corpus, but Word2Vec treats each occurrence as an independent prediction task. What if we could leverage this global co-occurrence information directly?

GloVe (Global Vectors for Word Representation) takes a different path. Developed by Pennington, Socher, and Manning at Stanford in 2014, GloVe starts with a co-occurrence matrix that captures how often words appear together across the entire corpus. It then factorizes this matrix to produce word vectors. The result: embeddings that encode both local context patterns and global corpus statistics.

This chapter develops GloVe from first principles. We'll see how the objective function emerges from a simple requirement that word vectors encode co-occurrence ratios, work through the weighted least squares formulation, and implement GloVe from scratch. By the end, you'll understand why GloVe achieves comparable results to Word2Vec despite taking a fundamentally different approach.

GloVe's key insight is simple but powerful: the ratio of co-occurrence probabilities encodes semantic relationships more reliably than raw probabilities. Consider the words "ice" and "steam." Both relate to water, but in different ways. How can we distinguish them?

Let's look at co-occurrence with probe words:

The raw probabilities $P(k \vert \text{ice})$ and $P(k \vert \text{steam})$ depend on many factors: how common each word is, the corpus domain, and so on. But the ratio tells a cleaner story:

• Large ratio: The probe word relates more to "ice" than "steam" (like "solid")
• Small ratio: The probe word relates more to "steam" than "ice" (like "gas")
• Ratio ≈ 1: The probe word relates equally to both (like "water") or to neither (like "fashion")

This ratio invariance is powerful. It factors out corpus-specific biases and isolates the semantic relationship we care about.

The ratios reveal clear discriminative patterns. "Solid" has a ratio far greater than 1 (approximately 8.6), indicating strong association with "ice" rather than "steam." Conversely, "gas" has a ratio well below 1 (approximately 0.03), showing the opposite relationship. Both "water" and "fashion" have ratios near 1, but for different reasons: "water" relates equally to both states, while "fashion" is irrelevant to either.

We've established that co-occurrence ratios encode semantic relationships. The next question is: how do we design word vectors that naturally capture these ratios? The answer comes through a derivation that starts with a simple requirement and, through a series of logical constraints, arrives at GloVe's objective function.

The derivation unfolds like a detective story. Each constraint eliminates possibilities, narrowing the space of potential solutions until only one sensible answer remains. By the end, the objective function won't feel like an arbitrary choice. It will feel inevitable.

Our starting point is the co-occurrence matrix $X$. This matrix is the foundation of everything GloVe does. Each entry $X_{ij}$ counts how often word $j$ appears within a context window of word $i$, accumulated across the entire corpus. From these raw counts, we can define probabilities:

where:

• $P_{ij}$: probability of word $j$ appearing in context of word $i$
• $X_{ij}$: co-occurrence count for words $i$ and $j$
• $X_i = \sum_k X_{ik}$: total co-occurrence count for word $i$

Now we can state our goal precisely. We want to learn word vectors $\mathbf{w}_i$ and context vectors $\tilde{\mathbf{w}}_k$ such that some function $F$ of these vectors recovers the co-occurrence ratio:

where:

• $\mathbf{w}_i, \mathbf{w}_j$: word vectors for the target words
• $\tilde{\mathbf{w}}_k$: context vector for the probe word
• $F$: function to be determined

This equation captures our key insight: the ratio of co-occurrence probabilities, the same ratio that distinguishes "ice" from "steam" via probe words like "solid" and "gas", should be computable from word vectors alone. The question is: what form must $F$ take?

The ratio $\frac{P_{ik}}{P_{jk}}$ fundamentally measures a contrast: how does word $i$'s relationship with $k$ differ from word $j$'s relationship with $k$? In vector spaces, the natural way to represent contrasts is through subtraction. When we compute $\mathbf{w}_i - \mathbf{w}_j$, we obtain a vector pointing from $j$ toward $i$, encoding everything that distinguishes them.

This suggests simplifying our function to depend on the difference:

Now $F$ takes two inputs: the difference vector $(\mathbf{w}_i - \mathbf{w}_j)$ and the context vector $\tilde{\mathbf{w}}_k$. This is already more constrained. $F$ doesn't need to handle three arbitrary vectors, just a difference and a context.

Look at the right-hand side: $\frac{P_{ik}}{P_{jk}}$ is a scalar, a single number. But our inputs are vectors, high-dimensional objects with many components. How do we combine two vectors to produce a single number?

The most natural choice is the dot product. The dot product $\mathbf{a} \cdot \mathbf{b}$ measures how aligned two vectors are: positive when they point similarly, negative when opposite, zero when perpendicular. It also has mathematical properties that will prove crucial shortly.

This gives us:

The function $F$ now operates on a scalar (the dot product) and produces another scalar. We've reduced the problem significantly.

Here's where the key insight emerges. First, let's expand the dot product using the distributive property:

where:

• $\mathbf{w}_i \cdot \tilde{\mathbf{w}}_k$: dot product between word $i$'s vector and context $k$'s vector
• $\mathbf{w}_j \cdot \tilde{\mathbf{w}}_k$: dot product between word $j$'s vector and context $k$'s vector

The left side is a difference of dot products. But look at the right side of our original equation: $\frac{P_{ik}}{P_{jk}}$ is a ratio of probabilities. We need a function $F$ that transforms differences into ratios.

Step 1: Identify the required property. Think about this algebraically: we need $F$ such that $F(a - b) = F(a)/F(b)$ for scalars $a$ and $b$. This is asking for a homomorphism from addition to multiplication, a function that converts additive structure into multiplicative structure.

Step 2: Recognize the exponential as the solution. There's essentially one continuous function with this property: the exponential. The key identity is:

This works because $e^{a-b} = e^a \cdot e^{-b} = e^a \cdot \frac{1}{e^b} = \frac{e^a}{e^b}$. The exponential naturally converts differences in the exponent into ratios in the output.

Step 3: Apply the exponential. Substituting into our equation from Constraint 2, we get:

Step 4: Separate the terms. Using the exponential property $e^{a-b} = e^a / e^b$, we can rewrite the left side:

Step 5: Match numerators and denominators. For this equality to hold for all word pairs $(i, j)$ and all probe words $k$, the numerators must be proportional and the denominators must be proportional. This means each individual term must satisfy:

where:

• $\exp(\mathbf{w}_i \cdot \tilde{\mathbf{w}}_k)$: exponential of the dot product between word and context vectors
• $P_{ik}$: probability of word $k$ appearing in context of word $i$
• $C$: a constant that may depend on $k$ but cancels when we take ratios

Step 6: Take logarithms. To make this more tractable, we take the natural logarithm of both sides:

This transforms the multiplicative relationship back into an additive one, which is easier to work with during optimization.

We're almost there. From Step 6 above, we have $\mathbf{w}_i \cdot \tilde{\mathbf{w}}_k = \log(P_{ik}) + \log(C)$. Now we substitute the definition of the conditional probability $P_{ik} = X_{ik} / X_i$:

Using the logarithm property $\log(a/b) = \log(a) - \log(b)$:

where:

• $X_{ik}$: co-occurrence count for word $i$ and context word $k$
• $X_i = \sum_k X_{ik}$: total co-occurrence count for word $i$ across all contexts
• $C$: the proportionality constant from Step 5 (may depend on $k$)

Now notice something important: $\log(X_i)$ depends only on word $i$, not on the context word $k$. This term captures how often word $i$ appears in the corpus overall, a frequency effect rather than a semantic relationship. Similarly, $\log(C)$ might depend only on $k$.

The solution is to absorb these word-specific terms into bias terms:

• Let $b_i = -\log(X_i) + \text{(other word-}i\text{-specific terms)}$
• Let $\tilde{b}_k = \log(C) + \text{(other context-}k\text{-specific terms)}$

This yields GloVe's core equation:

where:

• $\mathbf{w}_i$: word vector for word $i$ (a $d$-dimensional vector capturing semantic content)
• $\tilde{\mathbf{w}}_k$: context vector for word $k$ (a $d$-dimensional vector capturing contextual role)
• $b_i$: bias term for word $i$ (a scalar absorbing overall frequency effects)
• $\tilde{b}_k$: bias term for context word $k$ (a scalar absorbing context-specific effects)
• $X_{ik}$: co-occurrence count (the observed data we're trying to predict)

This equation has a clear interpretation. The dot product $\mathbf{w}_i \cdot \tilde{\mathbf{w}}_k$ measures the semantic compatibility between word $i$ and context $k$. The biases adjust for how common each word is overall. Together, they should predict the logarithm of how often we actually observe the pair together.

This is GloVe's core equation: the dot product of word and context vectors, plus biases, should equal the log co-occurrence count.

We've derived that word vectors should satisfy $\mathbf{w}_i \cdot \tilde{\mathbf{w}}_k + b_i + \tilde{b}_k = \log(X_{ik})$. But this is an idealized equation. In practice, no finite-dimensional embedding can perfectly satisfy it for every word pair. We need to frame this as an optimization problem: find the vectors and biases that come as close as possible to satisfying the equation across all pairs.

The journey from ideal equation to practical objective reveals important design decisions. A naive formulation encounters serious problems, and solving them leads to GloVe's distinctive weighted least squares approach.

The most straightforward optimization minimizes squared error:

where:

• $J$: the objective function (total loss) to minimize
• $\mathbf{w}_i \cdot \tilde{\mathbf{w}}_j + b_i + \tilde{b}_j$: our model's prediction for word pair $(i, j)$
• $\log(X_{ij})$: the target log co-occurrence count
• The sum is over all word pairs $(i, j)$ in the vocabulary

This says: for every word pair, measure how far our prediction deviates from the target log co-occurrence, square it (so positive and negative errors contribute equally), and sum across all pairs. Standard least squares.

But this formulation has critical flaws:

1. Zero counts are catastrophic: Many word pairs never co-occur in any corpus. "Quantum" and "umbrella" might never appear together. For these pairs, $X_{ij} = 0$, and $\log(0) = -\infty$. The objective becomes undefined.

2. Not all pairs deserve equal attention: A co-occurrence count of 1 million reflects a strong, statistically reliable signal. A count of 1 might be noise, a single accidental co-occurrence. Yet naive least squares weights them identically.

3. Rare pairs can dominate: If rare word pairs have large errors (which they often do, being noisy), they can disproportionately influence training, pulling embeddings away from configurations that would serve common words well.

These problems demand a more thoughtful objective.

GloVe's solution is to introduce a weighting function $f(X_{ij})$ that modulates how much each word pair contributes to the objective. This function is designed to satisfy three requirements:

1. Zero weight for zero counts: When $X_{ij} = 0$, set $f(0) = 0$. This pair simply doesn't contribute. We never try to predict $\log(0)$.

2. Increasing weight with frequency: Pairs that co-occur more often provide more reliable statistics. The function should increase with $X_{ij}$, giving more weight to confident observations.

3. Bounded influence: Extremely common word pairs (like "the" with almost everything) shouldn't completely dominate training. The weight should eventually plateau.

The function that GloVe adopts balances these requirements:

where:

• $x$: co-occurrence count $X_{ij}$ for a particular word pair
• $x_{\max}$: cutoff parameter (typically 100), beyond which the weight saturates
• $\alpha$: exponent (typically 0.75), controlling how quickly weight grows with frequency

Why this form works. The function has two regimes:

1. Below the cutoff ($x < x_{\max}$): The weight $(x / x_{\max})^\alpha$ grows sublinearly with co-occurrence count. The exponent $\alpha = 0.75$ (less than 1) ensures that doubling the count doesn't double the weight. Instead, it increases by a factor of $2^{0.75} \approx 1.68$. This dampens the influence of very frequent pairs while still giving them more weight than rare pairs.

2. At or above the cutoff ($x \geq x_{\max}$): The weight caps at 1.0. This prevents the most frequent word pairs (like "the" with common words) from completely dominating the training signal.

The weighting shows clear progression: a count of 1 receives weight 0.18, while a count of 50 gets 0.65. Once the count reaches 100 (the x_max threshold), the weight caps at 1.0 and stays there for higher counts. This sublinear scaling ($\alpha = 0.75$) means common word pairs contribute meaningfully to training without completely dominating rare but informative pairs.

Combining the core equation with the weighting function:

where:

• $J$: objective function to minimize
• $V$: vocabulary size
• $f(X_{ij})$: weighting function for word pair $(i, j)$
• $\mathbf{w}_i$: word vector for word $i$
• $\tilde{\mathbf{w}}_j$: context vector for word $j$
• $b_i, \tilde{b}_j$: bias terms
• $X_{ij}$: co-occurrence count

This is a weighted least squares problem: find vectors and biases that minimize the weighted squared error between predicted and actual log co-occurrences.

Before training GloVe, we must construct the co-occurrence matrix from a corpus. This preprocessing step represents a fundamental difference from Word2Vec: while Word2Vec processes training pairs on-the-fly during optimization, GloVe separates statistics gathering from model training. We scan the corpus once to build the co-occurrence matrix, then train on these precomputed counts.

This separation has important implications. The matrix construction phase is embarrassingly parallel, since each document can be processed independently. Once complete, the training phase operates on a fixed set of statistics, making it more predictable and easier to tune. The tradeoff is memory: we must store the entire matrix (though sparse representations help enormously).

The core question is: what exactly should we count? A word pair $(i, j)$ "co-occurs" when word $j$ appears within a context window of word $i$. But not all co-occurrences are equal. GloVe uses distance-weighted counting, where closer words contribute more to the co-occurrence count than distant ones.

The rationale is linguistic: words immediately adjacent typically have stronger relationships than words at the edges of a context window. In the phrase "the quick brown fox," "quick" and "brown" are more closely related than "the" and "fox," even though both pairs fall within a five-word window.

Specifically, if words $i$ and $j$ are separated by $d$ positions, we add $\frac{1}{d}$ to $X_{ij}$:

where:

• $d$: the distance in positions between words $i$ and $j$ (1 for adjacent words, 2 for one word apart, etc.)
• $X_{ij}$: the co-occurrence count being accumulated

Adjacent words (distance 1) contribute a full count of 1.0. Words two positions apart contribute 0.5. At the edge of a window of size 5, the contribution is just 0.2. This inverse-distance weighting encodes the intuition that proximity correlates with semantic relevance.

The matrix shows typical characteristics of word co-occurrence data. Even with this tiny corpus, the sparsity is substantial, as most word pairs never appear together within a context window. The highest co-occurrence counts involve "the," which appears frequently throughout the corpus. The distance-weighted counting produces fractional values: a count of 1.50 indicates two co-occurrences at different distances (e.g., adjacent words contribute 1.0, while words two positions apart contribute 0.5).

The co-occurrence matrix is nearly symmetric: $X_{ij} \approx X_{ji}$. For undirected context windows (looking both left and right), it's exactly symmetric. In practice, we often symmetrize the matrix by averaging:

where:

• $X'_{ij}$: the symmetrized co-occurrence count
• $X_{ij}$: count of word $j$ appearing in context of word $i$
• $X_{ji}$: count of word $i$ appearing in context of word $j$

This symmetrization ensures that the relationship between words $i$ and $j$ is treated the same regardless of which word is considered the "center" word.

For large vocabularies, the co-occurrence matrix is extremely sparse. A 100,000-word vocabulary produces a 10-billion-entry matrix, but most entries are zero. Efficient implementations use sparse matrix formats, storing only non-zero entries.

For this small vocabulary, the sparse format provides modest savings. The real benefit emerges at scale: a 100,000-word vocabulary would require approximately 80 GB for a dense matrix (100,000² × 8 bytes), while the sparse representation stores only non-zero entries, typically a few hundred megabytes. This difference makes large-scale GloVe training feasible on commodity hardware.

Stepping back from the implementation details reveals a deeper perspective on what GloVe is doing. The objective function we derived places GloVe squarely in the family of matrix factorization methods, the same family that includes techniques like Singular Value Decomposition (SVD) and Latent Semantic Analysis (LSA). Understanding this connection illuminates both why GloVe works and how it relates to classical dimensionality reduction.

Consider the equation we derived:

This equation holds for every word pair $(i, j)$. If we stack all word vectors into a matrix $W$ and all context vectors into a matrix $\tilde{W}$, we can write this as a matrix equation.

Let's define a target matrix $M$ where each entry $M_{ij} = \log(X_{ij})$ for non-zero co-occurrences. Then GloVe approximately factorizes this log-count matrix:

where:

• $M$: log co-occurrence matrix of size $V \times V$, where $M_{ij} = \log(X_{ij})$
• $W$: matrix of word vectors of size $V \times d$, where row $i$ is $\mathbf{w}_i^T$
• $\tilde{W}$: matrix of context vectors of size $V \times d$, where row $j$ is $\tilde{\mathbf{w}}_j^T$
• $W \tilde{W}^T$: the matrix product, where entry $(i, j)$ equals $\mathbf{w}_i \cdot \tilde{\mathbf{w}}_j$
• $\mathbf{b}$: column vector of word biases of size $V \times 1$
• $\tilde{\mathbf{b}}$: column vector of context biases of size $V \times 1$
• $\mathbf{1}$: column vector of ones of size $V \times 1$
• $\mathbf{b} \mathbf{1}^T$: outer product creating a $V \times V$ matrix where row $i$ has all entries equal to $b_i$
• $\mathbf{1} \tilde{\mathbf{b}}^T$: outer product creating a $V \times V$ matrix where column $j$ has all entries equal to $\tilde{b}_j$

This is a form of weighted, biased matrix factorization. The key insight is dimensional: the original matrix $M$ is $V \times V$, potentially enormous for large vocabularies. The factorization represents it as the product of much smaller matrices: $W$ and $\tilde{W}$ are both $V \times d$, where $d$ (typically 50-300) is vastly smaller than $V$ (potentially hundreds of thousands).

This compression is exactly what we want. The low-rank structure forces the model to discover patterns. It can't store the full matrix, so it must learn generalizable representations that explain many co-occurrences with few parameters. The resulting vectors $\mathbf{w}_i$ capture the essential semantic content of words, distilled from millions of co-occurrence observations.

The weighting function $f(X_{ij})$ makes this a weighted factorization, prioritizing accurate reconstruction of reliable observations. The biases make it biased (in the technical sense), allowing frequency effects to be absorbed separately from semantic content.

This matrix factorization perspective connects GloVe to classical methods like Latent Semantic Analysis (LSA), which factorizes term-document matrices using Singular Value Decomposition (SVD). Key differences:

GloVe inherits the global perspective of matrix factorization while adding neural-network-style training flexibility.

With the objective function and co-occurrence matrix defined, we can train GloVe using stochastic gradient descent. Unlike neural networks with complex layer compositions, GloVe's gradient computation is straightforward. The objective is a simple weighted sum of squared errors, and each term depends on only four parameters (two vectors and two biases).

The training process iterates through non-zero entries of the co-occurrence matrix, computing gradients and updating parameters. Because each word pair's contribution is independent, the computation parallelizes naturally across CPU cores or GPU threads.

Understanding the gradients illuminates how learning proceeds. For a single word pair $(i, j)$ with co-occurrence count $X_{ij}$, the contribution to the objective is:

where:

• $J_{ij}$: loss contribution from word pair $(i, j)$
• $f(X_{ij})$: weighting function that prioritizes frequent co-occurrences
• $\mathbf{w}_i \cdot \tilde{\mathbf{w}}_j + b_i + \tilde{b}_j$: our model's prediction
• $\log(X_{ij})$: the target value we want to predict

To derive the gradients, let's introduce cleaner notation that separates the prediction, target, and error:

• $\hat{y}_{ij} = \mathbf{w}_i \cdot \tilde{\mathbf{w}}_j + b_i + \tilde{b}_j$ (the model's prediction)
• $y_{ij} = \log(X_{ij})$ (the target value)
• $e_{ij} = \hat{y}_{ij} - y_{ij}$ (the prediction error)

The objective for this pair becomes $J_{ij} = f(X_{ij}) \cdot e_{ij}^2$. This is a weighted squared error: we penalize the squared prediction error, but modulate the penalty by the weight $f(X_{ij})$.

Step 1: Apply the chain rule. To find the gradient with respect to any parameter $\theta$, we use the chain rule. Since $J_{ij} = f(X_{ij}) \cdot e_{ij}^2$ and $f(X_{ij})$ is a constant (it depends on the data, not the parameters):

Since $e_{ij} = \hat{y}_{ij} - y_{ij}$ and $y_{ij}$ is a constant, we have $\frac{\partial e_{ij}}{\partial \theta} = \frac{\partial \hat{y}_{ij}}{\partial \theta}$:

Step 2: Compute partial derivatives of the prediction. Since $\hat{y}_{ij} = \mathbf{w}_i \cdot \tilde{\mathbf{w}}_j + b_i + \tilde{b}_j$, we can compute how the prediction changes with each parameter:

• $\frac{\partial \hat{y}_{ij}}{\partial \mathbf{w}_i} = \tilde{\mathbf{w}}_j$ (the derivative of $\mathbf{w}_i \cdot \tilde{\mathbf{w}}_j$ with respect to $\mathbf{w}_i$ is $\tilde{\mathbf{w}}_j$)
• $\frac{\partial \hat{y}_{ij}}{\partial \tilde{\mathbf{w}}_j} = \mathbf{w}_i$ (the derivative of $\mathbf{w}_i \cdot \tilde{\mathbf{w}}_j$ with respect to $\tilde{\mathbf{w}}_j$ is $\mathbf{w}_i$)
• $\frac{\partial \hat{y}_{ij}}{\partial b_i} = 1$ (the derivative of $b_i$ with respect to itself is 1)
• $\frac{\partial \hat{y}_{ij}}{\partial \tilde{b}_j} = 1$ (the derivative of $\tilde{b}_j$ with respect to itself is 1)

Step 3: Substitute to get the complete gradients. Plugging these partial derivatives into our chain rule formula:

Interpreting the gradients. Notice the symmetry: the gradient for word vectors involves the context vectors, and vice versa. This makes intuitive sense. To improve the prediction for the pair $(i, j)$, we adjust $\mathbf{w}_i$ in the direction of $\tilde{\mathbf{w}}_j$ (or opposite, if we're overshooting the target). The magnitude of the adjustment depends on:

1. The error $e_{ij}$: larger errors lead to larger updates
2. The weight $f(X_{ij})$: frequent co-occurrences receive larger updates

The biases have particularly simple gradients: just the weighted error, with no vector component. They act as global adjustments, shifting predictions up or down for all contexts involving word $i$ or context $j$.

Training iterates over all non-zero co-occurrence pairs, updating embeddings using gradient descent. The original GloVe implementation uses AdaGrad, an adaptive learning rate method that helps with the highly varying frequencies of word pairs.