GloVe: Global Vectors for Word Representation

GloVeLink Copied

Word2Vec learns embeddings through local context windows, predicting surrounding words one pair at a time. This approach works well but ignores a fundamental insight: some word relationships are global properties of the corpus. The word "ice" might appear near "cold" millions of times across a billion-word corpus, but Word2Vec treats each occurrence as an independent prediction task. What if we could leverage this global co-occurrence information directly?

GloVe (Global Vectors for Word Representation) takes a different path. Developed by Pennington, Socher, and Manning at Stanford in 2014, GloVe starts with a co-occurrence matrix that captures how often words appear together across the entire corpus. It then factorizes this matrix to produce word vectors. The result: embeddings that encode both local context patterns and global corpus statistics.

This chapter develops GloVe from first principles. We'll see how the objective function emerges from a simple requirement that word vectors encode co-occurrence ratios, work through the weighted least squares formulation, and implement GloVe from scratch. By the end, you'll understand why GloVe achieves comparable results to Word2Vec despite taking a fundamentally different approach.

The Insight: Co-occurrence Ratios Reveal MeaningLink Copied

GloVe's key insight is simple but powerful: the ratio of co-occurrence probabilities encodes semantic relationships more reliably than raw probabilities. Consider the words "ice" and "steam." Both relate to water, but in different ways. How can we distinguish them?

Let's look at co-occurrence with probe words:

The raw probabilities $P(k \mid \text{ice})$ and $P(k \mid \text{steam})$ depend on many factors: how common each word is, the corpus domain, and so on. But the ratio tells a cleaner story:

• Large ratio: The probe word relates more to "ice" than "steam" (like "solid")
• Small ratio: The probe word relates more to "steam" than "ice" (like "gas")
• Ratio ≈ 1: The probe word relates equally to both (like "water") or to neither (like "fashion")

This ratio invariance is powerful. It factors out corpus-specific biases and isolates the semantic relationship we care about.

The ratios reveal clear discriminative patterns. "Solid" has a ratio far greater than 1 (approximately 8.6), indicating strong association with "ice" rather than "steam." Conversely, "gas" has a ratio well below 1 (approximately 0.03), showing the opposite relationship. Both "water" and "fashion" have ratios near 1, but for different reasons: "water" relates equally to both states, while "fashion" is irrelevant to either.

From Ratios to Vectors: Deriving the ObjectiveLink Copied

We've established that co-occurrence ratios encode semantic relationships. The next question is: how do we design word vectors that naturally capture these ratios? The answer comes through a derivation that starts with a simple requirement and, through a series of logical constraints, arrives at GloVe's objective function.

The derivation unfolds like a detective story. Each constraint eliminates possibilities, narrowing the space of potential solutions until only one sensible answer remains. By the end, the objective function won't feel like an arbitrary choice. It will feel inevitable.

Setting Up the ProblemLink Copied

Our starting point is the co-occurrence matrix $X$. This matrix is the foundation of everything GloVe does. Each entry $X_{ij}$ counts how often word $j$ appears within a context window of word $i$, accumulated across the entire corpus. From these raw counts, we can define probabilities:

where:

• $P_{ij}$: probability of word $j$ appearing in context of word $i$
• $X_{ij}$: co-occurrence count for words $i$ and $j$
• $X_i = \sum_k X_{ik}$: total co-occurrence count for word $i$

Now we can state our goal precisely. We want to learn word vectors $\mathbf{w}_i$ and context vectors $\tilde{\mathbf{w}}_k$ such that some function $F$ of these vectors recovers the co-occurrence ratio:

where:

• $\mathbf{w}_i, \mathbf{w}_j$: word vectors for the target words
• $\tilde{\mathbf{w}}_k$: context vector for the probe word
• $F$: function to be determined

This equation captures our key insight: the ratio of co-occurrence probabilities, the same ratio that distinguishes "ice" from "steam" via probe words like "solid" and "gas", should be computable from word vectors alone. The question is: what form must $F$ take?

Constraint 1: Vector Differences Encode ContrastsLink Copied

The ratio $\frac{P_{ik}}{P_{jk}}$ fundamentally measures a contrast: how does word $i$'s relationship with $k$ differ from word $j$'s relationship with $k$? In vector spaces, the natural way to represent contrasts is through subtraction. When we compute $\mathbf{w}_i - \mathbf{w}_j$, we obtain a vector pointing from $j$ toward $i$, encoding everything that distinguishes them.

This suggests simplifying our function to depend on the difference:

Now $F$ takes two inputs: the difference vector $(\mathbf{w}_i - \mathbf{w}_j)$ and the context vector $\tilde{\mathbf{w}}_k$. This is already more constrained. $F$ doesn't need to handle three arbitrary vectors, just a difference and a context.

Constraint 2: Producing a Scalar from VectorsLink Copied

Look at the right-hand side: $\frac{P_{ik}}{P_{jk}}$ is a scalar, a single number. But our inputs are vectors, high-dimensional objects with many components. How do we combine two vectors to produce a single number?

The most natural choice is the dot product. The dot product $\mathbf{a} \cdot \mathbf{b}$ measures how aligned two vectors are: positive when they point similarly, negative when opposite, zero when perpendicular. It also has mathematical properties that will prove crucial shortly.

This gives us:

The function $F$ now operates on a scalar (the dot product) and produces another scalar. We've reduced the problem significantly.

Constraint 3: The Exponential EmergesLink Copied

Here's where the key insight emerges. Expand the dot product:

The left side is a difference of dot products. The right side of our equation is a ratio of probabilities. We need a function $F$ that transforms differences into ratios.

Think about this algebraically: we need $F$ such that $F(a - b) = F(a)/F(b)$ for scalars $a$ and $b$. This is asking for a homomorphism from addition to multiplication, a function that converts additive structure into multiplicative structure.

There's essentially one continuous function with this property: the exponential. Since $e^{a-b} = e^a / e^b$, the exponential naturally converts differences in the exponent into ratios in the output.

Applying this insight:

Using the exponential property, we can separate this into:

For this to hold for all word pairs, each individual term must satisfy:

for some constant $C$ that may depend on $k$ but cancels in the ratio. Taking logarithms of both sides:

Arriving at the Core EquationLink Copied

We're almost there. Substituting the definition $P_{ik} = X_{ik} / X_i$:

Now notice something important: $\log(X_i)$ depends only on word $i$, not on the context word $k$. This term captures how often word $i$ appears in the corpus overall, a frequency effect rather than a semantic relationship. Similarly, $\log(C)$ might depend only on $k$.

The solution is to absorb these word-specific terms into bias terms:

• Let $b_i = -\log(X_i) + \text{(other word-}i\text{-specific terms)}$
• Let $\tilde{b}_k = \log(C) + \text{(other context-}k\text{-specific terms)}$

This yields GloVe's core equation:

where:

• $\mathbf{w}_i$: word vector for word $i$ (captures semantic content)
• $\tilde{\mathbf{w}}_k$: context vector for word $k$ (captures contextual role)
• $b_i$: bias term for word $i$ (absorbs overall frequency effects)
• $\tilde{b}_k$: bias term for context word $k$ (absorbs context-specific effects)
• $X_{ik}$: co-occurrence count (the observed data)

This equation has a clear interpretation. The dot product $\mathbf{w}_i \cdot \tilde{\mathbf{w}}_k$ measures the semantic compatibility between word $i$ and context $k$. The biases adjust for how common each word is overall. Together, they should predict the logarithm of how often we actually observe the pair together.

This is GloVe's core equation: the dot product of word and context vectors, plus biases, should equal the log co-occurrence count.

The GloVe Objective FunctionLink Copied

We've derived that word vectors should satisfy $\mathbf{w}_i \cdot \tilde{\mathbf{w}}_k + b_i + \tilde{b}_k = \log(X_{ik})$. But this is an idealized equation. In practice, no finite-dimensional embedding can perfectly satisfy it for every word pair. We need to frame this as an optimization problem: find the vectors and biases that come as close as possible to satisfying the equation across all pairs.

The journey from ideal equation to practical objective reveals important design decisions. A naive formulation encounters serious problems, and solving them leads to GloVe's distinctive weighted least squares approach.

Naive Least Squares (and Its Problems)Link Copied

The most straightforward optimization minimizes squared error:

This says: for every word pair, measure how far our prediction deviates from the target log co-occurrence, square it (so positive and negative errors contribute equally), and sum across all pairs. Standard least squares.

But this formulation has critical flaws:

1. Zero counts are catastrophic: Many word pairs never co-occur in any corpus. "Quantum" and "umbrella" might never appear together. For these pairs, $X_{ij} = 0$, and $\log(0) = -\infty$. The objective becomes undefined.

2. Not all pairs deserve equal attention: A co-occurrence count of 1 million reflects a strong, statistically reliable signal. A count of 1 might be noise, a single accidental co-occurrence. Yet naive least squares weights them identically.

3. Rare pairs can dominate: If rare word pairs have large errors (which they often do, being noisy), they can disproportionately influence training, pulling embeddings away from configurations that would serve common words well.

These problems demand a more thoughtful objective.

The Weighting FunctionLink Copied

GloVe's solution is to introduce a weighting function $f(X_{ij})$ that modulates how much each word pair contributes to the objective. This function is designed to satisfy three requirements:

1. Zero weight for zero counts: When $X_{ij} = 0$, set $f(0) = 0$. This pair simply doesn't contribute. We never try to predict $\log(0)$.

2. Increasing weight with frequency: Pairs that co-occur more often provide more reliable statistics. The function should increase with $X_{ij}$, giving more weight to confident observations.

3. Bounded influence: Extremely common word pairs (like "the" with almost everything) shouldn't completely dominate training. The weight should eventually plateau.

The function that GloVe adopts balances these requirements:

where:

• $x$: co-occurrence count $X_{ij}$
• $x_{\max}$: cutoff parameter (typically 100)
• $\alpha$: exponent (typically 0.75)

The weighting shows clear progression: a count of 1 receives weight 0.18, while a count of 50 gets 0.65. Once the count reaches 100 (the x_max threshold), the weight caps at 1.0 and stays there for higher counts. This sublinear scaling ($\alpha = 0.75$) means common word pairs contribute meaningfully to training without completely dominating rare but informative pairs.

The Complete ObjectiveLink Copied

Combining the core equation with the weighting function:

where:

• $J$: objective function to minimize
• $V$: vocabulary size
• $f(X_{ij})$: weighting function for word pair $(i, j)$
• $\mathbf{w}_i$: word vector for word $i$
• $\tilde{\mathbf{w}}_j$: context vector for word $j$
• $b_i, \tilde{b}_j$: bias terms
• $X_{ij}$: co-occurrence count

This is a weighted least squares problem: find vectors and biases that minimize the weighted squared error between predicted and actual log co-occurrences.

Building the Co-occurrence MatrixLink Copied

Before training GloVe, we must construct the co-occurrence matrix from a corpus. This preprocessing step represents a fundamental difference from Word2Vec: while Word2Vec processes training pairs on-the-fly during optimization, GloVe separates statistics gathering from model training. We scan the corpus once to build the co-occurrence matrix, then train on these precomputed counts.

This separation has important implications. The matrix construction phase is embarrassingly parallel, since each document can be processed independently. Once complete, the training phase operates on a fixed set of statistics, making it more predictable and easier to tune. The tradeoff is memory: we must store the entire matrix (though sparse representations help enormously).

Defining Co-occurrenceLink Copied

The core question is: what exactly should we count? A word pair $(i, j)$ "co-occurs" when word $j$ appears within a context window of word $i$. But not all co-occurrences are equal. GloVe uses distance-weighted counting, where closer words contribute more to the co-occurrence count than distant ones.

The rationale is linguistic: words immediately adjacent typically have stronger relationships than words at the edges of a context window. In the phrase "the quick brown fox," "quick" and "brown" are more closely related than "the" and "fox," even though both pairs fall within a five-word window.

Specifically, if words $i$ and $j$ are separated by $d$ positions, we add $1/d$ to $X_{ij}$. Adjacent words (distance 1) contribute a full count of 1.0. Words two positions apart contribute 0.5. At the edge of a window of size 5, the contribution is just 0.2. This inverse-distance weighting encodes the intuition that proximity correlates with semantic relevance.

The matrix shows typical characteristics of word co-occurrence data. Even with this tiny corpus, the sparsity is substantial, as most word pairs never appear together within a context window. The highest co-occurrence counts involve "the," which appears frequently throughout the corpus. The distance-weighted counting produces fractional values: a count of 1.50 indicates two co-occurrences at different distances (e.g., adjacent words contribute 1.0, while words two positions apart contribute 0.5).

Symmetry and Sparse StorageLink Copied

The co-occurrence matrix is nearly symmetric: $X_{ij} \approx X_{ji}$. For undirected context windows (looking both left and right), it's exactly symmetric. In practice, we often symmetrize the matrix by averaging: $X'_{ij} = (X_{ij} + X_{ji}) / 2$.

For large vocabularies, the co-occurrence matrix is extremely sparse. A 100,000-word vocabulary produces a 10-billion-entry matrix, but most entries are zero. Efficient implementations use sparse matrix formats, storing only non-zero entries.

For this small vocabulary, the sparse format provides modest savings. The real benefit emerges at scale: a 100,000-word vocabulary would require approximately 80 GB for a dense matrix (100,000² × 8 bytes), while the sparse representation stores only non-zero entries, typically a few hundred megabytes. This difference makes large-scale GloVe training feasible on commodity hardware.

Relationship to Matrix FactorizationLink Copied

Stepping back from the implementation details reveals a deeper perspective on what GloVe is doing. The objective function we derived places GloVe squarely in the family of matrix factorization methods, the same family that includes techniques like Singular Value Decomposition (SVD) and Latent Semantic Analysis (LSA). Understanding this connection illuminates both why GloVe works and how it relates to classical dimensionality reduction.

Consider the equation we derived:

This equation holds for every word pair. If we stack all word vectors into a matrix $W$ and all context vectors into a matrix $\tilde{W}$, we can write this as a matrix equation. Define $M_{ij} = \log(X_{ij})$ for non-zero entries. Then GloVe approximately factorizes this log-count matrix:

where:

• $M$: log co-occurrence matrix where $M_{ij} = \log(X_{ij})$
• $W$: matrix of word vectors ($V \times d$)
• $\tilde{W}$: matrix of context vectors ($V \times d$)
• $\mathbf{b}$: word biases ($V \times 1$)
• $\tilde{\mathbf{b}}$: context biases ($V \times 1$)
• $\mathbf{1}$: vector of ones ($V \times 1$)

This is a form of weighted, biased matrix factorization. The key insight is dimensional: the original matrix $M$ is $V \times V$, potentially enormous for large vocabularies. The factorization represents it as the product of much smaller matrices: $W$ and $\tilde{W}$ are both $V \times d$, where $d$ (typically 50-300) is vastly smaller than $V$ (potentially hundreds of thousands).

This compression is exactly what we want. The low-rank structure forces the model to discover patterns. It can't store the full matrix, so it must learn generalizable representations that explain many co-occurrences with few parameters. The resulting vectors $\mathbf{w}_i$ capture the essential semantic content of words, distilled from millions of co-occurrence observations.

The weighting function $f(X_{ij})$ makes this a weighted factorization, prioritizing accurate reconstruction of reliable observations. The biases make it biased (in the technical sense), allowing frequency effects to be absorbed separately from semantic content.

Connection to Classical MethodsLink Copied

This matrix factorization perspective connects GloVe to classical methods like Latent Semantic Analysis (LSA), which factorizes term-document matrices using Singular Value Decomposition (SVD). Key differences:

GloVe inherits the global perspective of matrix factorization while adding neural-network-style training flexibility.

Training GloVeLink Copied

With the objective function and co-occurrence matrix defined, we can train GloVe using stochastic gradient descent. Unlike neural networks with complex layer compositions, GloVe's gradient computation is straightforward. The objective is a simple weighted sum of squared errors, and each term depends on only four parameters (two vectors and two biases).

The training process iterates through non-zero entries of the co-occurrence matrix, computing gradients and updating parameters. Because each word pair's contribution is independent, the computation parallelizes naturally across CPU cores or GPU threads.

Gradient DerivationLink Copied

Understanding the gradients illuminates how learning proceeds. For a single word pair $(i, j)$ with co-occurrence count $X_{ij}$, the contribution to the objective is:

To derive the gradients, let's introduce cleaner notation:

• $\hat{y}_{ij} = \mathbf{w}_i \cdot \tilde{\mathbf{w}}_j + b_i + \tilde{b}_j$ (prediction)
• $y_{ij} = \log(X_{ij})$ (target)
• $e_{ij} = \hat{y}_{ij} - y_{ij}$ (error)

The objective for this pair becomes $J_{ij} = f(X_{ij}) \cdot e_{ij}^2$. Using the chain rule, the gradient with respect to any parameter $\theta$ is:

Since $\hat{y}_{ij} = \mathbf{w}_i \cdot \tilde{\mathbf{w}}_j + b_i + \tilde{b}_j$, the partial derivatives of the prediction are simple:

• $\frac{\partial \hat{y}_{ij}}{\partial \mathbf{w}_i} = \tilde{\mathbf{w}}_j$ (the context vector)
• $\frac{\partial \hat{y}_{ij}}{\partial \tilde{\mathbf{w}}_j} = \mathbf{w}_i$ (the word vector)
• $\frac{\partial \hat{y}_{ij}}{\partial b_i} = 1$
• $\frac{\partial \hat{y}_{ij}}{\partial \tilde{b}_j} = 1$

Substituting these, we get the complete gradients:

Notice the symmetry: the gradient for word vectors involves the context vectors, and vice versa. This makes intuitive sense. To improve the prediction for the pair $(i, j)$, we adjust $\mathbf{w}_i$ in the direction of $\tilde{\mathbf{w}}_j$ (or opposite, if we're overshooting the target). The magnitude of the adjustment depends on the error $e_{ij}$ and the weight $f(X_{ij})$.

The biases have particularly simple gradients: just the weighted error, with no vector component. They act as global adjustments, shifting predictions up or down for all contexts involving word $i$ or context $j$.

Training LoopLink Copied

Training iterates over all non-zero co-occurrence pairs, updating embeddings using gradient descent. The original GloVe implementation uses AdaGrad, an adaptive learning rate method that helps with the highly varying frequencies of word pairs.

The substantial loss reduction indicates the model successfully learned to predict log co-occurrence counts. The weighted least squares objective prioritizes high-frequency pairs, so the final embeddings should capture the dominant co-occurrence patterns in the corpus. The remaining loss reflects both inherent noise in co-occurrence statistics and the capacity limitations of a 20-dimensional embedding space.

The Role of Bias TermsLink Copied

GloVe includes bias terms $b_i$ and $\tilde{b}_j$ in its objective. These aren't just mathematical conveniences; they play a crucial role in capturing word frequency effects.

What Biases CaptureLink Copied

Consider a very frequent word like "the." It co-occurs with nearly every word in the vocabulary, leading to high co-occurrence counts across the board. Without biases, the model would try to explain these high counts through the embedding: "the" would need a large vector that has high dot product with everything.

Biases absorb this frequency effect. The bias $b_i$ captures "how often word $i$ tends to co-occur in general." A word like "the" has a high bias, explaining its high co-occurrence counts without distorting its embedding.

The pattern is clear: words with higher total co-occurrence counts tend to have larger learned biases. The most frequent word in the corpus shows the highest combined bias, absorbing its tendency to co-occur with many different words. This prevents the embedding vectors from needing unnaturally large magnitudes to explain high co-occurrence counts, keeping the learned semantic relationships clean and interpretable.

The comparison is striking. Biases show strong correlation with frequency (left panel), while embedding norms show weaker correlation (right panel). This confirms that biases are doing their job: absorbing frequency effects so that the embedding vectors can focus on encoding semantic content.

Combining Word and Context VectorsLink Copied

GloVe learns two sets of vectors: word embeddings $\mathbf{w}_i$ and context embeddings $\tilde{\mathbf{w}}_i$. Unlike Word2Vec, where these play asymmetric roles, GloVe's objective is symmetric in $i$ and $j$. This means $\mathbf{w}_i$ and $\tilde{\mathbf{w}}_i$ carry similar information.

The original GloVe paper recommends combining them:

where:

• $\mathbf{v}_i$: final word vector for word $i$
• $\mathbf{w}_i$: word embedding from the word matrix $W$
• $\tilde{\mathbf{w}}_i$: context embedding from the context matrix $\tilde{W}$

This combination often produces better embeddings than either matrix alone. Intuitively, it averages out noise and captures complementary aspects of word meaning.

GloVe vs Word2Vec: Key DifferencesLink Copied

GloVe and Word2Vec both produce high-quality word embeddings, but they take fundamentally different approaches. Understanding these differences helps you choose the right method for your application.

Training ParadigmLink Copied

Practical ConsiderationsLink Copied

Memory usage: GloVe requires storing the co-occurrence matrix (sparse but potentially large), while Word2Vec can stream data from disk.

Training speed: GloVe often converges faster because it uses global statistics. Word2Vec may need multiple passes over the corpus.

Parallelization: GloVe's matrix operations parallelize naturally on GPUs. Word2Vec's sequential updates are harder to parallelize, though implementations use negative sampling tricks.

Rare words: Both struggle with rare words, but GloVe's explicit co-occurrence counts make the signal clearer.

Empirical PerformanceLink Copied

The original GloVe paper reported competitive results on word analogy and similarity tasks:

The GloVe paper showed substantial improvements over Word2Vec on these benchmarks, particularly for semantic analogies. However, subsequent research has demonstrated that careful hyperparameter tuning can close much of this gap. The choice between GloVe and Word2Vec often depends on practical considerations like data availability, memory constraints, and training infrastructure, rather than absolute quality differences.

Training GloVe EfficientlyLink Copied

Real-world GloVe training involves millions of words and billions of co-occurrences. Several techniques make this feasible.

Sparse Storage and IterationLink Copied

The co-occurrence matrix is extremely sparse. A 100,000-word vocabulary has 10 billion potential entries, but only millions are non-zero. Efficient training iterates only over non-zero entries.

AdaGrad OptimizationLink Copied

GloVe uses AdaGrad, which adapts the learning rate for each parameter based on its historical gradients. Parameters updated frequently (like biases for common words) receive smaller updates, while rare parameters receive larger updates.

where:

• $\theta_t$: parameter at time $t$
• $\eta$: base learning rate
• $G_t$: sum of squared gradients up to time $t$
• $g_t$: current gradient
• $\epsilon$: small constant for numerical stability

ParallelizationLink Copied

Each co-occurrence pair can be processed independently (up to synchronization of embedding updates). GPU implementations parallelize across thousands of pairs simultaneously.

Evaluating GloVe EmbeddingsLink Copied