Word Embeddings: Word2Vec, GloVe and Vector Semantics › GloVe: Global Vectors from Matrix Factorization

GloVe (Global Vectors), introduced by Pennington et al. in 2014, takes a fundamentally different approach from Word2Vec. Instead of learning from local context windows one at a time, GloVe leverages global co-occurrence statistics across the entire corpus. This shift from local to global information allows GloVe to capture more nuanced word relationships.

The breakthrough insight behind GloVe is that word relationships can be captured through ratios of co-occurrence probabilities, not just the probabilities themselves. Let's see why this matters.

Consider the words "ice" and "steam". Both are forms of water, so they might both co-occur with words like "water" and "temperature". But their differences become clear when we look at ratios. "Ice" co-occurs much more frequently with "solid" than "steam" does, while "steam" co-occurs much more with "gas" than "ice" does. The ratios reveal these relationships:

$$\frac{P(\text{solid} | \text{ice})}{P(\text{solid} | \text{steam})} \gg 1$$ $$\frac{P(\text{gas} | \text{ice})}{P(\text{gas} | \text{steam})} \ll 1$$

where:

• $P(\text{solid} | \text{ice})$: the conditional probability of observing "solid" in the context of "ice"
• $P(\text{solid} | \text{steam})$: the conditional probability of observing "solid" in the context of "steam"
• The ratio $\gg 1$ means the probability is much greater for "ice" than "steam"
• Similarly, $P(\text{gas} | \text{ice}) / P(\text{gas} | \text{steam}) \ll 1$ means "gas" is much more likely with "steam" than "ice"

The first ratio is much greater than 1 because "solid" appears far more often with "ice" than with "steam". The second ratio is much less than 1 because "gas" appears far more often with "steam" than with "ice". These ratios capture semantic relationships that raw co-occurrence counts might miss.

GloVe's goal is to learn embeddings such that vector differences capture these co-occurrence ratios. If we can represent words as vectors where $\mathbf{w}_{\text{ice}} - \mathbf{w}_{\text{steam}}$ captures the difference in how they relate to other words, we've encoded semantic relationships directly in the embedding space.

GloVe starts by building a co-occurrence matrix $X$ that captures how often words appear together across the entire corpus. This matrix is fundamentally different from Word2Vec's approach: instead of processing text one window at a time, we first collect all co-occurrence statistics, then learn embeddings from this global view.

The matrix $X$ has dimensions $V \times V$ (vocabulary size by vocabulary size), where $X_{ij}$ counts how often word $j$ appears in the context of word $i$. The context is typically defined by a symmetric window around each word. For example, with window size 2, we count co-occurrences within $\pm 2$ words on either side.

Here's how we build it: slide a window across the corpus, and for each center word $i$ and context word $j$ within the window, increment $X_{ij}$. We also apply distance weighting: words closer to the center word contribute more to the count than words farther away. This reflects the intuition that nearby words are more relevant.

The resulting matrix is typically very sparse: most word pairs never co-occur in the corpus. But the non-zero entries contain rich information. Words that appear in similar contexts will have similar rows (or columns) in this matrix, which is exactly what we want embeddings to capture.

Now we face the core challenge: how do we convert this co-occurrence matrix into dense word embeddings? This is a matrix factorization problem. We want to find low-dimensional vectors (embeddings) such that their interactions approximate the co-occurrence statistics.

GloVe's approach is elegant: learn embeddings such that the dot product between word vectors approximates the logarithm of co-occurrence count. Specifically, we want:

$$\mathbf{w}_i^T \tilde{\mathbf{w}}_j + b_i + \tilde{b}_j \approx \log X_{ij}$$

where:

• $\mathbf{w}_i \in \mathbb{R}^d$: the embedding vector for word $i$ when it appears as a center word, where $d$ is the embedding dimension
• $\tilde{\mathbf{w}}_j \in \mathbb{R}^d$: the embedding vector for word $j$ when it appears as a context word
• $\mathbf{w}_i^T \tilde{\mathbf{w}}_j$: the dot product (scalar) between the two embedding vectors
• $b_i \in \mathbb{R}$: a bias term for word $i$ that captures its overall frequency as a center word
• $\tilde{b}_j \in \mathbb{R}$: a bias term for word $j$ that captures its overall frequency as a context word
• $X_{ij}$: the co-occurrence count (how many times word $j$ appears in the context of word $i$)
• $\log X_{ij}$: the natural logarithm of the co-occurrence count (using $\log$ base $e$)

The goal is to learn embeddings such that their dot product (plus biases) approximates the logarithm of the co-occurrence count. Taking the logarithm turns multiplicative relationships into additive ones, which are easier to model with linear operations like dot products.

Why the logarithm? Co-occurrence counts can vary over many orders of magnitude. The word "the" might co-occur with thousands of words millions of times, while rare word pairs might co-occur only once. The logarithm compresses this range, making the optimization more stable. It also has a nice mathematical property: ratios become differences under the logarithm, which aligns with our goal of capturing relationships through vector differences.

Why two embedding matrices? Like Word2Vec, GloVe learns separate embeddings for words as center words ($\mathbf{w}_i$) and as context words ($\tilde{\mathbf{w}}_j$). This asymmetry allows the model to capture different aspects of word meaning depending on role. After training, we typically combine them (often by addition: $\mathbf{w}_i + \tilde{\mathbf{w}}_i$) or use just the center word embeddings.

To learn these embeddings, GloVe minimizes a weighted least squares objective:

$$J = \sum_{i,j=1}^{V} f(X_{ij}) (\mathbf{w}_i^T \tilde{\mathbf{w}}_j + b_i + \tilde{b}_j - \log X_{ij})^2$$

where:

• $J$: the GloVe objective function (loss) to minimize
• $V$: the vocabulary size
• $i, j \in \{1, 2, \ldots, V\}$: indices for words in the vocabulary
• $X_{ij}$: the co-occurrence count for word pair $(i, j)$
• $f(X_{ij})$: the weighting function (defined below) that determines the importance of each word pair
• $\mathbf{w}_i^T \tilde{\mathbf{w}}_j + b_i + \tilde{b}_j$: the predicted log co-occurrence from the embeddings
• $\log X_{ij}$: the actual log co-occurrence from the corpus
• The squared term $(\mathbf{w}_i^T \tilde{\mathbf{w}}_j + b_i + \tilde{b}_j - \log X_{ij})^2$: the squared error between predicted and actual log co-occurrence

The squared term: $(\mathbf{w}_i^T \tilde{\mathbf{w}}_j + b_i + \tilde{b}_j - \log X_{ij})^2$ measures how well our predicted log co-occurrence (from embeddings) matches the actual log co-occurrence (from the matrix). We want this difference to be small for all word pairs.

The weighting function $f(X_{ij})$: This is crucial. Not all co-occurrence counts are equally important. The weighting function:

$$f(x) = \begin{cases} (x/x_{\max})^{\alpha} & \text{if } x < x_{\max} \\ 1 & \text{otherwise} \end{cases}$$

where:

• $f(x)$: the weighting function that assigns importance to co-occurrence count $x$
• $x$: the co-occurrence count $X_{ij}$ for a word pair
• $x_{\max}$: the maximum count threshold (typically $x_{\max} = 100$)
• $\alpha$: the exponent controlling the downweighting strength (typically $\alpha = 0.75$)
• When $x = 0$: $f(0) = 0$ (zero weight for non-co-occurring pairs)
• When $0 < x < x_{\max}$: $f(x) = (x/x_{\max})^{\alpha}$ (gradually increasing weight)
• When $x \geq x_{\max}$: $f(x) = 1$ (full weight for frequent pairs)

This weighting function serves three purposes:

1. Handles sparsity: Zero co-occurrences ($X_{ij} = 0$) get zero weight, so we don't try to fit the vast number of word pairs that never appear together. This is important because the matrix is mostly zeros.

2. Downweights very frequent co-occurrences: Words like "the" co-occur with almost everything. Without downweighting, these high-frequency pairs would dominate the objective function, preventing the model from learning subtle relationships between content words.

3. Prevents rare co-occurrences from dominating: Very rare word pairs (co-occurring once or twice) might be noise rather than meaningful relationships. The weighting function gives them less influence than moderate-frequency pairs, which are more reliable.

The exponent $\alpha = 0.75$ is a hyperparameter that controls the strength of downweighting. Values closer to 1 give more weight to frequent pairs; values closer to 0 give more uniform weighting. The value 0.75 was found empirically to work well across different corpora.

Why least squares? The squared error is a natural choice for regression problems. It's differentiable, has nice optimization properties, and penalizes large errors more than small ones (quadratic penalty). This encourages the model to get the most important relationships right, even if it sacrifices accuracy on rare or noisy pairs.

GloVe's matrix factorization approach has several advantages over Word2Vec's local window method:

1. Efficient use of statistics: All co-occurrence information is used simultaneously. Word2Vec processes one window at a time, potentially missing global patterns. GloVe sees the full picture from the start.

2. Explicit relationship to co-occurrence: The objective function directly relates embeddings to co-occurrence statistics. This makes the model more interpretable: we can understand why certain embeddings emerge by looking at the co-occurrence matrix.

3. Better handling of rare words: By using global statistics, GloVe can learn better representations for rare words that might not appear in enough local windows for Word2Vec to learn effectively.

However, GloVe requires storing the full co-occurrence matrix, which can be memory-intensive for very large vocabularies (though the matrix is sparse and can be stored efficiently). In practice, both Word2Vec and GloVe produce high-quality embeddings, with GloVe often performing slightly better on semantic analogy tasks.