GloVe: Word Embeddings via Co-Occurrence Matrix

Word2Vec and its variants (Skip-gram, CBOW) learn embeddings by training a neural network to predict words from their context. The embeddings emerge as a byproduct of that prediction task: the network never directly sees the corpus statistics it is using; it only sees individual context windows one at a time. But there is an older, more direct tradition in NLP for capturing word relationships: count how often words co-occur in the same context across a large corpus, then factorize the resulting co-occurrence matrix into dense vectors. This approach has deep roots in Latent Semantic Analysis from the 1990s, but it fell out of favor when prediction-based methods demonstrated cleaner results on analogy benchmarks.

GloVe (Global Vectors for Word Representation), introduced by Pennington, Socher, and Manning at Stanford in 2014, bridges these two traditions. It brings the global, corpus-wide statistics of matrix factorization methods together with the efficient, scalable training machinery of prediction-based methods like Word2Vec. The result is a model that is theoretically cleaner than Word2Vec, often faster to train, and produces word vectors of comparable or better quality on standard benchmarks.

The key insight behind GloVe is deceptively simple: word co-occurrence ratios carry more signal than raw co-occurrence counts. If "ice" and "steam" are both related to "water", their co-occurrence probabilities with "water" will be similar. But if you look at how each co-occurs with "solid", "ice" wins by a large margin. The ratio $P(\text{solid} | \text{ice}) / P(\text{solid} | \text{steam})$ reveals something meaningful: "solid" is much more associated with ice than with steam. GloVe's objective function is designed to directly capture these ratios in the geometry of the embedding space.

This chapter derives GloVe's objective function from first principles, shows every algebraic step without shortcuts, explains the weighted least squares formulation and its design choices, connects GloVe to classical matrix factorization and PMI, covers the bias terms and weighting function, and compares GloVe to Word2Vec in terms of training dynamics, performance, and practical use. We then implement GloVe from scratch and explore pretrained vectors.

Before deriving the GloVe objective, we need to understand what the input data looks like, why raw counts fall short, and what we are really trying to capture.

Given a corpus and a context window size $W$, the co-occurrence matrix $\mathbf{X}$ has entry $X_{ij}$ counting how often word $j$ appears in the context of word $i$ within the window. If we treat both left and right context symmetrically (which is the standard choice), this matrix is symmetric: $X_{ij} = X_{ji}$.

The window size matters more than it might seem. With a narrow window of size 2, a word's context consists of its immediate neighbors, so syntactic roles and local collocation patterns dominate. "Run" and "runs" will be close, and "fast" and "quickly" will be similar because they modify similar verbs. With a wide window of size 10, a word's context includes anything in the surrounding sentence or even beyond, so topically related words cluster together even if they never appear immediately adjacent. "Doctor" and "hospital" will be similar because they share many sentence-level co-occurrence partners.

From the raw counts we can compute conditional probabilities. Let $X_i = \sum_k X_{ik}$ be the total count of all words appearing in the context of word $i$. Then:

where:

• $X_{ij}$: the number of times word $j$ appears in the context of word $i$
• $X_i = \sum_k X_{ik}$: the total number of context word occurrences for word $i$ (the row sum)

This is the probability of word $j$ appearing in the context of word $i$. It normalizes the raw count by the total context budget for word $i$, so frequent words like "the" do not automatically dominate.

The core motivation for GloVe comes from observing that ratios of conditional probabilities are more informative than the probabilities themselves. Consider two words: "ice" ($i$) and "steam" ($j$). We look at how each co-occurs with a probe word $k$:

The ratios tell a far cleaner story than the raw probabilities:

• "solid" has a high ratio: it is related to ice but not steam
• "gas" has a low ratio: it is related to steam but not ice
• "water" has a ratio near 1: it is related to both equally
• "fashion" has a ratio near 1: it is related to neither

A good word vector model should encode this structure. The ratio $P(k|i)/P(k|j)$ discriminates relevant probe words from irrelevant ones far better than either probability alone.

Why are ratios so much better than raw probabilities? Consider what raw probabilities tell you. $P(\text{solid} | \text{ice}) = 0.00019$ is a tiny number that, taken alone, gives almost no information: it just says "solid" is somewhat rare in any context. But the ratio 8.9 immediately reveals a relationship: solid is about 9 times more associated with ice than with steam. The ratio normalizes away the baseline frequency of the probe word, leaving only the relative discriminative power.

This insight motivates GloVe's entire design. The model is built to encode these ratios in the geometry of the embedding space. When the vectors for "ice" and "steam" are represented as points in a high-dimensional space, their relative positions should reflect the pattern you see in the table above: "solid" should be much closer to "ice" than to "steam", "gas" should be the reverse, and "water" should sit equidistant.

The log scale makes the contrast clear. "Solid" has a ratio of about 8.9 (strongly associated with ice), "gas" has a ratio of about 0.085 (strongly associated with steam), while "water" and "fashion" cluster near 1. Ratios near 1 mean the probe word does not discriminate between ice and steam.

The derivation starts from the ratios and works backward to a tractable optimization objective. Each step follows necessarily from a design choice, so the final formula is not arbitrary: it is the natural consequence of the ratio insight combined with a small set of reasonable constraints.

We want word vectors $\vec{w}_i$, $\vec{w}_j$, and $\vec{w}_k$ such that some function $F$ of these vectors recovers the ratio:

where:

• $\vec{w}_i$: the word vector for word $i$ (the first target word)
• $\vec{w}_j$: the word vector for word $j$ (the second target word we are comparing against)
• $\vec{w}_k$: the vector for the probe word $k$
• $P_{ik} = P(k \mid i) = X_{ik}/X_i$: the conditional probability of seeing $k$ in the context of $i$

We want this function to depend on the difference $\vec{w}_i - \vec{w}_j$, because we are comparing two target words and the comparison should be captured as a vector difference. The simplest way to combine a difference vector with a third vector is via a dot product:

Here $\tilde{\vec{w}}_k$ is a separate vector for word $k$ when it is a context word, distinct from $\vec{w}_k$ when $k$ is the target word. This distinction is standard in embedding models: each word plays two roles (as a target and as a context), and the two roles are learned separately. Using separate vectors for the two roles avoids conflating them and is a key part of why the derivation works out cleanly.

We need $F$ to satisfy the ratio decomposition property. When we expand the ratio $P_{ik}/P_{jk}$, we are taking the ratio of two quantities that each involve a single target word and the probe word $k$. So we need:

This is a functional equation: we need a function $F$ such that $F(a - b) = F(a)/F(b)$ for all $a$ and $b$. The exponential function satisfies this exactly, since:

This is the only continuous function (up to a constant multiplier) that converts subtraction into division. The uniqueness here is reassuring: we are not choosing $\exp$ arbitrarily, it is forced by the design requirement.

Plugging in $F = \exp$ and setting the result equal to the true ratio $P_{ik}/P_{jk}$ gives us, for each individual word pair $(i, k)$:

Taking the natural logarithm of both sides:

The term $\log X_i$ depends only on word $i$, not on the context word $k$. It represents word $i$'s overall frequency as a context center: a word that appears in many contexts has a large $X_i$. We can absorb this word-frequency effect into a scalar bias $b_i$ learned during training. Similarly, by symmetry, we add a context bias $\tilde{b}_k$ for word $k$'s frequency as a context word:

where:

• $b_i$: a scalar bias for word $i$ that absorbs its context frequency $\log X_i$
• $\tilde{b}_k$: a scalar bias for word $k$ in its role as a context word

The context bias $\tilde{b}_k$ is not strictly required by the derivation, but it restores symmetry. Because $X_{ik} = X_{ki}$ for symmetric windows, both the target role and the context role of each word should have their own frequency-absorbing bias. Adding $\tilde{b}_k$ makes the formulation symmetric in the roles of target and context.

This equation is the core relationship GloVe trains for. We want, for every word pair $(i, j)$, the model's prediction $\vec{w}_i^T \tilde{\vec{w}}_j + b_i + \tilde{b}_j$ to equal the corpus-derived target $\log X_{ij}$.

We want our word vectors and biases to satisfy $\vec{w}_i^T \tilde{\vec{w}}_j + b_i + \tilde{b}_j \approx \log X_{ij}$ for all $(i, j)$ pairs. The natural formulation is a least squares objective over all $V^2$ word pairs:

But this treats all co-occurrence pairs equally, and that creates two serious problems. First, rare co-occurrences (small $X_{ij}$) are statistically noisy. If "quasar" and "kumquat" appeared together once in a 100-billion-token corpus, that single co-occurrence is almost certainly an accident, not evidence of a semantic relationship. Yet the unweighted objective would treat this pair as equally important as "cat" co-occurring with "dog" 10,000 times. Second, most entries of $X$ are zero (words that never co-occurred), and $\log 0$ is undefined, so the plain objective is not even computable.

The solution is to weight each term by a function $f(X_{ij})$ that satisfies four properties:

1. Zero co-occurrences get zero weight, so $\log 0$ never appears in the sum
2. Rare co-occurrences get lower weight, reducing the influence of statistical noise
3. Very frequent co-occurrences do not get excessive weight (stop words like "the" co-occur with almost everything, but without a semantic relationship)
4. The function is continuous and non-decreasing

This gives the final GloVe objective:

where:

• $f(X_{ij})$: the weighting function that scales each term's contribution by the reliability of the co-occurrence count
• $\vec{w}_i^T \tilde{\vec{w}}_j + b_i + \tilde{b}_j$: the model's prediction of $\log X_{ij}$
• $\log X_{ij}$: the target value derived from the corpus

This is the weighted least squares objective that GloVe minimizes. Because $f(0) = 0$, the sum effectively runs only over pairs where $X_{ij} > 0$. For large vocabularies this is a small fraction of all $V^2$ pairs, making the sum tractable.

The weighting function $f$ must satisfy the properties listed above. The original GloVe paper proposes a specific polynomial form with a plateau:

where:

• $x$: the co-occurrence count $X_{ij}$ for the pair being weighted
• $x_{\max}$: the saturation threshold at which weight reaches its maximum of 1 (default: 100)
• $\alpha$: the exponent controlling how quickly weight grows with count (default: $3/4$)

Let us unpack each design choice carefully, because together they handle several failure modes at once.

The saturation threshold $x_{\max}$: Co-occurrences above $x_{\max}$ receive full weight 1. Below $x_{\max}$, weight scales polynomially from 0 to 1. This prevents very common co-occurrences, like "the" appearing with essentially every content word, from dominating the objective simply by virtue of their frequency. Without this cap, function words would drown out the signal from semantically rich but less frequent pairs. With the cap, every pair above $x_{\max}$ contributes equally to the objective, and only pairs below the threshold get downweighted.

The exponent $\alpha = 3/4$: This value is not arbitrary. It is the same exponent used in Word2Vec's negative sampling unigram distribution. Values close to 1 give nearly linear weighting (doubling the count roughly doubles the weight), while values close to 0 give more uniform weighting (the weight barely changes with count). The $3/4$ exponent is a middle ground: it penalizes very rare co-occurrences without being too aggressive. Empirically, it produces better embeddings than either $\alpha = 1$ (too linear) or $\alpha = 0.5$ (too flat). The $3/4$ value was tuned in the original paper across multiple downstream tasks.

Zero weight for zero co-occurrences: Because $f(0) = 0$, pairs where $X_{ij} = 0$ never appear in the sum. This elegantly sidesteps the $\log 0$ problem. You do not need to add a smoothing constant or handle zeros as a special case; the weighting function simply ignores them. In practice, the sum runs over the nonzero entries of $\mathbf{X}$, which is exactly the set of pairs that contain real information.

One subtlety: the choice of $x_{\max} = 100$ means that only pairs co-occurring at least 100 times receive full weight. For smaller corpora, you may need to lower this threshold. If your corpus has fewer than a million tokens, a threshold of 10 or 20 may be more appropriate.

The weighting function makes GloVe robust to two failure modes: rare co-occurrences that are dominated by noise, and very common co-occurrences that tend to reflect syntactic accidents rather than semantic relationships. The polynomial shape below $x_{\max}$ means the transition from zero weight to full weight is smooth, which helps with gradient-based optimization.

The bias terms $b_i$ and $\tilde{b}_j$ play an important and often underappreciated role in making the objective symmetric and absorbing word-frequency effects.

Recall that the derivation gave us $\vec{w}_i^T \tilde{\vec{w}}_j = \log X_{ij} - \log X_i$. The term $\log X_i$ (the log total context count for word $i$) depends only on how frequent word $i$ is overall. Very frequent words like "the" have large $X_i$ regardless of the specific context word. If we did not include a bias term, this frequency effect would be absorbed into the dot product, corrupting the relational structure the vectors are supposed to capture. The bias $b_i$ absorbs this word-frequency effect, freeing the dot product to represent purely relational information.

Similarly, $\tilde{b}_j$ absorbs the frequency effect for word $j$ in its role as a context word. Together, $b_i + \tilde{b}_j$ accounts for both the target and context word's marginal frequencies, letting the dot product $\vec{w}_i^T \tilde{\vec{w}}_j$ focus on the residual: how much more or less often $i$ and $j$ co-occur than we would predict from their individual frequencies alone. This residual is precisely what PMI measures, as we will see in the next section.

The bias terms are simple scalars, so they add very few parameters (only $2V$ additional values) while providing a significant modeling benefit. In practice, the biases converge quickly during training because they are solving a simple regression problem: fit the word's average log-count across all its context partners.

An elegant consequence of the bias terms is that the objective behaves consistently when we swap the roles of target and context words. In the co-occurrence matrix $\mathbf{X}$, entry $X_{ij}$ counts word $j$ in word $i$'s context, and $X_{ji}$ counts word $i$ in word $j$'s context. For symmetric windows, $X_{ij} = X_{ji}$. The GloVe objective with bias terms naturally accommodates this symmetry: both $(\vec{w}_i^T \tilde{\vec{w}}_j + b_i + \tilde{b}_j - \log X_{ij})^2$ and $(\vec{w}_j^T \tilde{\vec{w}}_i + b_j + \tilde{b}_i - \log X_{ji})^2$ are minimized simultaneously.

GloVe's symmetric design means that after training, the word vector $\vec{w}_i$ and its context vector $\tilde{\vec{w}}_i$ capture similar but not identical information. They have learned from the same co-occurrence data, just from different roles in the objective. A common and empirically validated practice is to use $\vec{w}_i + \tilde{\vec{w}}_i$ as the final word representation, combining the two complementary views for slightly improved performance. This is analogous to ensemble averaging: the two vectors have seen the same data from different angles, and combining them reduces variance in the final representation.

Why do the two vectors differ at all if they see the same data? Because the objective is not symmetric in the parameters, even though it is symmetric in the data. The gradients with respect to $\vec{w}_i$ depend on $\tilde{\vec{w}}_j$, and vice versa. So although both vectors converge toward representations that encode the same underlying statistics, the optimization path is different, and the resulting vectors are slightly different directions in the embedding space. Summing them gives a representation that combines both views.

GloVe has a deep connection to classical matrix factorization methods, particularly Latent Semantic Analysis (LSA) and Pointwise Mutual Information (PMI) factorization. Understanding this connection reveals why GloVe works and what it computes.

LSA applies Singular Value Decomposition (SVD) to a term-document matrix, factorizing it into two lower-rank matrices whose rows become word vectors. A related approach applies SVD to the Pointwise Mutual Information (PMI) matrix, where:

where:

• $P(i, j) = X_{ij}/N$: the joint probability of words $i$ and $j$ co-occurring
• $P(i) = X_i/N$: the marginal probability of word $i$ appearing as a context center
• $P(j) = X_j/N$: the marginal probability of word $j$ appearing as a context word
• $N = \sum_{ij} X_{ij}$: total co-occurrence count across the entire corpus

PMI measures how much more often words $i$ and $j$ co-occur than we would expect if they were statistically independent. A high positive PMI means the two words appear together far more often than chance; a negative PMI means they avoid each other. In practice, Positive PMI (PPMI) is often used, which clips negative values to zero, because negative PMI values are unreliable for sparse data.

Factorizing the PMI matrix with SVD produces word vectors that, like GloVe, capture semantic relationships. But SVD has two practical limitations. First, zero entries in the PMI matrix (words that never co-occurred) create numerical issues: the log of zero is undefined, so zero entries must either be excluded or replaced with a large negative value. Second, SVD weights all entries equally, giving as much importance to the noisy "the" co-occurring with "antidisestablishmentarianism" once as to "ice" co-occurring with "cold" a thousand times. Neither limitation is easy to fix within the SVD framework.

GloVe's target $\log X_{ij}$ is related to PMI. Starting from the PMI definition:

Rearranging:

The terms $\log X_i$, $\log X_j$, and $\log N$ are all word-frequency effects and constant offsets. The bias terms $b_i$ and $\tilde{b}_j$ absorb these terms during training. So the dot product $\vec{w}_i^T \tilde{\vec{w}}_j$ learns to approximate:

In other words, GloVe implicitly factorizes a shifted PMI matrix. The shift is $-\log N$, a constant that ensures the target values are centered, not unlike how Shifted Positive PMI (SPPMI) works in the PMI literature.

This connection, established formally by Levy and Goldberg (2014) shortly after the GloVe paper, is a key theoretical contribution. It shows that prediction-based methods (Word2Vec) and count-based methods (GloVe, PMI factorization) are not fundamentally different: they are all learning to factorize different variants of the PMI matrix. The apparent dichotomy between the two traditions dissolves under this unified view.

A naive log-count factorization via SVD suffers from the two problems described above: zero entries and uniform weighting. GloVe's weighted least squares formulation addresses both:

Zero entries are handled by the weighting function: $f(0) = 0$ means zero co-occurrences contribute nothing to the objective, so the $\log 0$ problem never arises.

Unequal reliability is handled by giving each pair a weight proportional to how informative its count is. Common, reliable pairs get weight close to 1; rare, noisy pairs get lower weight. This is exactly the kind of reweighting that SVD cannot express, because SVD treats all entries of the matrix symmetrically.

GloVe can also be thought of as a noise-robust version of PMI factorization. By using a polynomial weighting scheme tuned for NLP statistics, GloVe extracts the signal from co-occurrence data more efficiently than either raw SVD or unweighted least squares.

Both GloVe and Word2Vec produce high-quality word embeddings, but they approach the problem from different angles and have distinct practical tradeoffs. Understanding these differences helps you choose between them and set appropriate expectations.

Word2Vec (Skip-gram with negative sampling, or SGNS) trains a binary classifier: given a target word and a context word, is this a real pair or a noise pair sampled from the unigram distribution? The model never explicitly sees co-occurrence counts; it learns from individual context windows presented one at a time during training. Each pass through the corpus updates the vectors for the words in each window.

GloVe works from aggregate statistics. It first computes the full co-occurrence matrix $\mathbf{X}$ by scanning the corpus once (or a few times for large corpora), then fits vectors to reproduce the log-count structure. No context windows are processed during the training phase; only the pre-computed counts matter.

This distinction has several practical consequences:

• GloVe training over the co-occurrence matrix is embarrassingly parallel: all word pairs are independent once the matrix is built, so GloVe can be parallelized across many CPUs or GPUs trivially
• Word2Vec training requires streaming through the corpus sequentially (or in large parallel chunks), maintaining state about which windows have been seen
• GloVe explicitly uses corpus-wide statistics; every training step uses global count information rather than a single local window
• For very large corpora (hundreds of billions of tokens), building the co-occurrence matrix requires significant memory, though sparse storage makes it feasible for vocabularies up to several million words

The training dynamics also differ. Word2Vec with negative sampling implicitly performs stochastic gradient descent over a noise-contrastive objective. GloVe with AdaGrad performs direct regression against a fixed target matrix. Word2Vec's implicit curriculum (frequently seen pairs get more gradient updates) is handled explicitly in GloVe through the weighting function.

Despite their different approaches, GloVe and Word2Vec produce qualitatively similar embeddings with comparable performance on standard downstream tasks. Both capture syntactic and semantic regularities and support the vector arithmetic that makes analogies like "king - man + woman = queen" work. The differences that do exist are modest and corpus-dependent:

• GloVe tends to train faster given a fixed co-occurrence matrix, because regression against a deterministic target converges quickly with AdaGrad
• Word2Vec is more memory-efficient during the training phase (no $V \times V$ matrix needed), though it requires the full corpus to be accessible
• GloVe's performance is sensitive to the window size used for counting, with larger windows producing more topically oriented embeddings
• On analogy benchmarks, both methods achieve 60-75% accuracy on standard evaluation sets, with small differences depending on corpus size and hyperparameter choices

The theoretical connection between the two methods (both factorize PMI variants) suggests that their performance should be similar in the limit of large corpora and optimal hyperparameters. Empirically, this is borne out. The practical choice between them often comes down to infrastructure rather than quality: GloVe is easier to parallelize and inspect, while Word2Vec integrates naturally into streaming training pipelines.

A third option worth mentioning is FastText, which extends the word embedding idea by decomposing words into character n-grams. This gives it a significant advantage for rare words and morphologically rich languages, but at the cost of more complex training. We will cover FastText in the next chapter.

For most practical NLP applications, using pretrained GloVe or Word2Vec vectors interchangeably gives similar results. The cases where the choice matters more are:

When training from scratch on a small corpus (<10M tokens), the difference between the two methods can be larger and less predictable. Empirical evaluation on your specific task and domain is the only reliable guide.

When memory is severely constrained, Word2Vec's streaming approach may be preferable because it never needs to store the full co-occurrence matrix.

When you need to inspect what the model learned, GloVe's explicit PMI factorization interpretation makes analysis more principled: you can compare the learned dot products directly to PMI values computed from the corpus.

GloVe minimizes the weighted least squares objective using AdaGrad, an adaptive gradient descent optimizer well suited to sparse, high-dimensional problems. The update rule for word vector $\vec{w}_i$ at step $t$ for a sampled pair $(i, j)$ proceeds as follows.

Step 1: Compute the residual (prediction error):

This is the signed error between the model's prediction and the target log co-occurrence count. A positive residual means the model predicts the pair is more similar than the data supports; a negative residual means the data shows more co-occurrence than the model currently encodes.

Step 2: Compute the gradient with respect to $\vec{w}_i$:

The weighting $f(X_{ij})$ scales the gradient: pairs with low weight contribute a smaller gradient, updating the vectors less aggressively for noisy co-occurrences. This links between the weighting function and the optimization: rare pairs not only contribute less to the final objective value, they also push the vectors less during each training step.

Step 3: Accumulate squared gradients (AdaGrad's adaptive memory):

AdaGrad maintains a separate accumulated squared gradient for each parameter dimension. Dimensions that receive large gradients accumulate large $G_i$ values.

Step 4: Update the word vector with an adaptive learning rate:

where:

• $\eta$: the global learning rate (typically 0.05)
• $G_i$: accumulated sum of squared gradients for word $i$ (element-wise)
• $\epsilon$: small constant for numerical stability (e.g., $10^{-8}$)

The $1/\sqrt{G_i}$ factor is AdaGrad's key feature. Dimensions that receive large gradients accumulate larger $G_i$, resulting in smaller effective learning rates. This stabilizes training when some word dimensions are updated far more frequently than others, which happens naturally in word embedding problems: high-frequency words update their vectors often while rare words update infrequently. The same updates apply symmetrically to $\tilde{\vec{w}}_j$, $b_i$, and $\tilde{b}_j$.

GloVe samples co-occurrence pairs $(i, j)$ in proportion to their weight $f(X_{ij})$, so high-weight pairs are seen more often during each epoch. Training typically runs for 50-100 epochs over all nonzero $(i, j)$ pairs.

AdaGrad was specifically designed for sparse learning problems, which is exactly the structure of GloVe training. Most words in a large vocabulary are rare: the top 1,000 words by frequency account for the majority of tokens, while the remaining 999,000 words appear only occasionally. In a single training epoch, common words like "the" and "of" will be involved in thousands of co-occurrence pairs, while rare technical terms may appear in only a handful.

Standard SGD with a fixed learning rate struggles with this imbalance: the learning rate must be small enough not to overshoot for common words, but this makes rare word updates almost imperceptibly small. AdaGrad solves this by maintaining per-parameter learning rates: rare word parameters accumulate small gradients and therefore keep large effective learning rates, while common word parameters accumulate large gradients and get smaller effective rates. The result is a more uniform convergence across the vocabulary.

The main drawback of AdaGrad is that the accumulated squared gradient $G_i$ only ever increases, so the effective learning rate only ever decreases. For long training runs, this can slow convergence to a crawl. More modern optimizers like Adam use an exponential moving average of squared gradients instead, which decays the memory of old gradients and allows the learning rate to recover. In practice, GloVe training with AdaGrad works well for the typical 50-100 epoch range before this becomes a significant issue.

The key hyperparameters for GloVe training interact in ways that are worth understanding.

Window size: The original paper uses a window of $W = 10$ (5 words on each side) for large corpora. Smaller windows (1-3) emphasize syntactic and local collocational relationships. Larger windows (10-20) emphasize topical and semantic relationships. In practice, a window of 5-10 is a reliable default for most applications. For tasks like POS tagging or parsing where local syntax matters, a narrower window often helps. For tasks like document similarity or topic modeling where broad semantics matter, a wider window is better.

Embedding dimensionality: Dimensions of 50-300 work well for most tasks. The original paper evaluated 50d, 100d, 200d, and 300d vectors and found that performance plateaus or slightly improves beyond 100d for most tasks, with diminishing returns above 300d. The choice involves a practical tradeoff: higher dimensions provide more expressive power but increase memory usage and can make downstream model training more expensive. For applications where the embedding is a fixed feature (not fine-tuned), 100d is often a good default.

Vocabulary size and minimum count: The co-occurrence matrix size scales as $O(V^2)$ in the worst case, but in practice most entries are zero. Discarding words that appear fewer than 5-10 times in the corpus reduces $V$ substantially while retaining nearly all useful semantic information. Words that appear fewer than 5 times simply do not have enough co-occurrence data to learn reliable vectors.

Corpus size: GloVe benefits substantially from large corpora. The original paper trained on Common Crawl (840 billion tokens) and Wikipedia (6 billion tokens). For smaller corpora (<100 million tokens), Word2Vec's online learning may generalize better because it can extract more signal from each individual context window.

Let's implement GloVe from scratch: build a co-occurrence matrix, define the weighted least squares objective, and train word vectors with PyTorch. We will use a small toy corpus to make the computation tractable, but every component scales directly to large vocabularies.

We start by tokenizing a corpus and computing the symmetric co-occurrence matrix within a fixed window. A common refinement is inverse-distance weighting: context words farther from the center contribute a fractional count ($1/\text{distance}$) rather than a full count. This gives more weight to immediate neighbors and less weight to distant context, capturing syntactic proximity more naturally.

The high sparsity is expected: most pairs of words in the vocabulary never appear in the same context window. Real-world vocabularies of 400,000 words would be even sparser, with far less than 1% of entries being nonzero.

The model has four learnable components per word: a word vector (when the word is the target), a context vector (when the word is in context), and bias scalars for each role.

The initialization of word and context vectors with small random values scaled by $1/\text{embed\_dim}$ follows the original GloVe paper. Initializing biases to zero is natural because the bias should start as a neutral correction that the optimizer adjusts as needed.

Note that x_max=10.0 is used here (rather than the paper's 100) to account for the tiny corpus. With only 12 sentences, co-occurrence counts rarely exceed 10, so a lower saturation threshold is appropriate.

Each training pair is an $(i, j)$ index pair, the log co-occurrence count $\log X_{ij}$, and the weight $f(X_{ij})$. The entire co-occurrence matrix is loaded into memory as a flat list of nonzero pairs, which is the standard efficient representation for sparse training data.

The log-scale plot reveals a common pattern in GloVe training: a steep initial drop followed by a much slower refinement phase. The initial drop corresponds to the vectors moving from random initialization to a rough approximation of the log co-occurrence structure. The slow refinement phase is the model fine-tuning the dot products to match the data more precisely, with AdaGrad's decreasing learning rates naturally slowing this phase down.

After training, we combine word and context vectors as the final representation. This is the averaging trick described earlier: summing the two complementary views of each word.

The cosine similarities reflect the semantic structure in the corpus. Words from the same semantic domain ("ice" and "cold", "cat" and "dog") should show higher similarity than cross-domain pairs ("ice" and "dog"). With only 12 sentences and 8 dimensions, the signal is weak but directionally correct.

With only 8 dimensions and a tiny corpus, the embeddings will not be perfect. But projecting to 2D with PCA lets us check whether semantically similar words cluster together.

In practice, you rarely train GloVe from scratch. Pretrained vectors from the Stanford NLP Group, trained on billions of tokens from Wikipedia and Common Crawl, are far more useful for most applications. The pretrained vectors have seen vocabulary and co-occurrence statistics that no small-corpus training run can match.

The analogy task works by vector arithmetic in the embedding space. "king - man + woman" computes the vector for "king", subtracts the vector offset between male and female (encoded as $\vec{w}_{\text{man}} - \vec{w}_{\text{woman}}$), and finds the nearest neighbor in vocabulary. The fact that "queen" emerges near the top of this list demonstrates that GloVe's objective successfully encodes semantic relationships as directional offsets in the embedding geometry.

Let's verify the original motivating insight: that GloVe embeddings encode co-occurrence ratios as geometric relationships.

Words like "solid" and "cold" show much higher cosine similarity to "ice" than to "steam" in the embedding space, while "water" is roughly equidistant. This is exactly the pattern predicted by the co-occurrence ratios in the paper's motivating table. The geometry of the learned space directly reflects the statistical structure of the corpus.

When using GloVe, either loading pretrained vectors or training from scratch, the key parameters to understand are:

• embed_dim: Embedding dimensionality. Values of 50 to 300 work well for most tasks. Larger dimensions capture more nuance but require more training data and memory. The Stanford pretrained vectors come in 50d, 100d, 200d, and 300d variants.
• window: Context window size for building the co-occurrence matrix. Larger windows (5 to 10) capture broad topical and semantic relationships. Smaller windows (1 to 3) capture syntactic patterns more tightly.
• x_max: The co-occurrence count at which the weighting function saturates to 1. Pairs with counts above this threshold receive full weight. The paper recommends 100 for large corpora, but smaller values suit smaller corpora.
• alpha: The exponent in the weighting function. The paper recommends 0.75, which was empirically validated across multiple analogy and similarity benchmarks.
• learning_rate: AdaGrad learning rate. The paper uses 0.05. AdaGrad's adaptive nature makes this less sensitive than in standard SGD.
• n_epochs: Number of training passes over all nonzero co-occurrence pairs. Typically 50 to 100 epochs is sufficient for convergence on large corpora.

GloVe's impact on NLP was substantial. When the paper appeared in 2014, it provided the first rigorous theoretical connection between count-based and prediction-based methods, helping unify two previously separate research traditions. Before GloVe, it was unclear whether the quality difference between Word2Vec and LSA-style matrix factorization was due to the model architecture, the training objective, the use of global versus local statistics, or some combination. GloVe's derivation showed that the key ingredient was encoding co-occurrence ratios rather than raw probabilities, and that prediction-based methods were implicitly doing something very similar. This theoretical clarity guided subsequent research.

Practically, pretrained GloVe vectors became (and remain) a standard baseline for many NLP tasks. Loading 50 to 300 dimensional GloVe vectors as input features improves performance on tasks like named entity recognition, sentiment analysis, text classification, and question answering without any task-specific embedding training. The vectors transfer well because they capture general English semantics from massive corpora, encoding both syntactic patterns (singular/plural, tense) and semantic relationships (synonyms, antonyms, analogies).

The impact was also pedagogical. GloVe's clean derivation from first principles made it possible to explain exactly why word embeddings work, not just demonstrate that they do. Every step of the derivation, from the ratio insight to the exponential functional form to the bias terms, follows logically. This clarity made GloVe a popular pedagogical tool in NLP courses and textbooks.

GloVe shares fundamental limitations with all static embedding methods. Each word gets exactly one vector regardless of context, so "bank" (financial institution) and "bank" (river bank) map to the same point in embedding space. GloVe cannot distinguish between these two senses. As we will explore in later chapters on contextual representations (ELMo, BERT, and their successors), this limitation becomes critical for language understanding tasks where meaning is highly context-dependent. A polysemous word like "bat" needs a different representation in "the bat hung from the ceiling" versus "she swung the bat", and a single static vector cannot capture this.

GloVe's reliance on pre-computed co-occurrence statistics also means that training cannot incorporate new data incrementally. Adding a new document to the corpus requires recomputing the affected rows of the co-occurrence matrix and retraining. Word2Vec's streaming nature is more amenable to online updates, which matters for applications where the corpus evolves over time, such as social media or news data.

The $V \times V$ co-occurrence matrix poses scalability challenges. For a vocabulary of one million words, even a sparse matrix representation can be very large. In practice, aggressive vocabulary pruning (discarding low-frequency words) is necessary. This means GloVe handles rare words poorly: words seen fewer than the minimum count threshold receive no embedding at all. FastText, covered in the next chapter, addresses this by representing words as bags of character n-grams, allowing it to construct vectors for unseen or rare words by composing their subword representations.

Finally, like Word2Vec, GloVe embeddings encode statistical biases present in the training corpus. Gender stereotypes, cultural biases, and historical associations are all faithfully reproduced in the embedding geometry. The parallelogram model for analogies can reveal these biases: "man is to programmer as woman is to homemaker" is a documented failure mode of embeddings trained on unfiltered web text. Research into debiasing word embeddings (projecting out gender subspaces, adjusting co-occurrence statistics) became an active area precisely because GloVe's theoretical transparency made the bias encoding mechanism easy to analyze. The same geometric structure that makes analogies work also makes biases geometrically accessible and measurable.

A subtler limitation is that GloVe treats all co-occurrence pairs independently. Two words that never co-occur directly but frequently share the same context (for example, "cat" and "feline" might rarely appear in the same sentence but both appear near "meow", "purr", and "fur") will have their similarity captured only transitively through their shared context partners. In practice this works well, but it means the model cannot explicitly represent transitive or higher-order relationships.

GloVe derives word embeddings by fitting vectors to reproduce the log co-occurrence statistics of a large corpus. The derivation follows from a single insight about ratios, and each step is logically necessary given the design requirements.

The key ideas are:

• Co-occurrence ratios carry more discriminative signal than raw probabilities, and GloVe's objective is built to encode these ratios in the geometry of the embedding space
• The weighted least squares objective $J = \sum_{i,j} f(X_{ij})(\vec{w}_i^T \tilde{\vec{w}}_j + b_i + \tilde{b}_j - \log X_{ij})^2$ directly fits vectors to log co-occurrence counts, with each term weighted by co-occurrence reliability
• The functional form $F = \exp$ is forced by the requirement that $F(a - b) = F(a)/F(b)$, connecting vector differences to probability ratios
• The weighting function $f(x) = \min(1, (x/x_{\max})^\alpha)$ downweights rare and zero co-occurrences, solving both the $\log 0$ problem and noise sensitivity
• Bias terms $b_i$ and $\tilde{b}_j$ absorb marginal word-frequency effects, letting the dot product capture pure relational structure
• GloVe implicitly factorizes a shifted PMI matrix, unifying count-based and prediction-based embedding methods under a single theoretical framework
• AdaGrad adapts per-parameter learning rates, handling the extreme frequency imbalance in natural language vocabularies
• Combining word and context vectors ($\vec{w}_i + \tilde{\vec{w}}_i$) as the final representation reduces variance and slightly improves quality
• Pretrained GloVe vectors trained on billions of tokens provide strong features for downstream NLP tasks without any task-specific embedding training

The next chapter on FastText extends the word embedding idea in a different direction: instead of learning one vector per word type, it decomposes words into character n-grams, enabling representations for out-of-vocabulary words and better handling of morphologically rich languages.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about GloVe word embeddings.