GloVe › Connection to Matrix Factorization

GloVe has a deep connection to classical matrix factorization methods, particularly Latent Semantic Analysis (LSA) and Pointwise Mutual Information (PMI) factorization. Understanding this connection reveals why GloVe works and what it computes.

LSA applies Singular Value Decomposition (SVD) to a term-document matrix, factorizing it into two lower-rank matrices whose rows become word vectors. A related approach applies SVD to the Pointwise Mutual Information (PMI) matrix, where:

$$\text{PMI}(i, j) = \log \frac{P(i, j)}{P(i) \cdot P(j)} = \log \frac{X_{ij} \cdot N}{X_i \cdot X_j}$$

where:

• $P(i, j) = X_{ij}/N$: the joint probability of words $i$ and $j$ co-occurring
• $P(i) = X_i/N$: the marginal probability of word $i$ appearing as a context center
• $P(j) = X_j/N$: the marginal probability of word $j$ appearing as a context word
• $N = \sum_{ij} X_{ij}$: total co-occurrence count across the entire corpus

PMI measures how much more often words $i$ and $j$ co-occur than we would expect if they were statistically independent. A high positive PMI means the two words appear together far more often than chance; a negative PMI means they avoid each other. In practice, Positive PMI (PPMI) is often used, which clips negative values to zero, because negative PMI values are unreliable for sparse data.

Factorizing the PMI matrix with SVD produces word vectors that, like GloVe, capture semantic relationships. But SVD has two practical limitations. First, zero entries in the PMI matrix (words that never co-occurred) create numerical issues: the log of zero is undefined, so zero entries must either be excluded or replaced with a large negative value. Second, SVD weights all entries equally, giving as much importance to the noisy "the" co-occurring with "antidisestablishmentarianism" once as to "ice" co-occurring with "cold" a thousand times. Neither limitation is easy to fix within the SVD framework.

GloVe's target $\log X_{ij}$ is related to PMI. Starting from the PMI definition:

$$\begin{aligned} \text{PMI}(i,j) &= \log \frac{X_{ij} \cdot N}{X_i \cdot X_j} \\ &= \log X_{ij} + \log N - \log X_i - \log X_j \end{aligned}$$

Rearranging:

$$\log X_{ij} = \text{PMI}(i,j) + \log X_i + \log X_j - \log N$$

The terms $\log X_i$, $\log X_j$, and $\log N$ are all word-frequency effects and constant offsets. The bias terms $b_i$ and $\tilde{b}_j$ absorb these terms during training. So the dot product $\vec{w}_i^T \tilde{\vec{w}}_j$ learns to approximate:

$$\vec{w}_i^T \tilde{\vec{w}}_j \approx \text{PMI}(i,j) - \log N$$

In other words, GloVe implicitly factorizes a shifted PMI matrix. The shift is $-\log N$, a constant that ensures the target values are centered, not unlike how Shifted Positive PMI (SPPMI) works in the PMI literature.

This connection, established formally by Levy and Goldberg (2014) shortly after the GloVe paper, is a key theoretical contribution. It shows that prediction-based methods (Word2Vec) and count-based methods (GloVe, PMI factorization) are not fundamentally different: they are all learning to factorize different variants of the PMI matrix. The apparent dichotomy between the two traditions dissolves under this unified view.

A naive log-count factorization via SVD suffers from the two problems described above: zero entries and uniform weighting. GloVe's weighted least squares formulation addresses both:

Zero entries are handled by the weighting function: $f(0) = 0$ means zero co-occurrences contribute nothing to the objective, so the $\log 0$ problem never arises.

Unequal reliability is handled by giving each pair a weight proportional to how informative its count is. Common, reliable pairs get weight close to 1; rare, noisy pairs get lower weight. This is exactly the kind of reweighting that SVD cannot express, because SVD treats all entries of the matrix symmetrically.

GloVe can also be thought of as a noise-robust version of PMI factorization. By using a polynomial weighting scheme tuned for NLP statistics, GloVe extracts the signal from co-occurrence data more efficiently than either raw SVD or unweighted least squares.