Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

GloVe › From Co-Occurrence Counts to Word Relationships

Michael BrenndoerferApril 10, 20256 min read

Language AI HandbookAll chapters

Part 4 of 67Word Embeddings

Chapter 25GloVe › From Co-Occurrence Counts to Word Relationships

Reading section 2 of 13 · From Co-Occurrence Counts to Word Relationships · 6 minSection 2 of 13 · 6 min

Section 2 of 13 · 6 min read

GloVe › From Co-Occurrence Counts to Word RelationshipsOne section · Sections in this chapter2 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Before deriving the GloVe objective, we need to understand what the input data looks like, why raw counts fall short, and what we are really trying to capture.

Building a Co-Occurrence MatrixLink Copied

Given a corpus and a context window size $W$ , the co-occurrence matrix $\mathbf{X}$ has entry $X_{ij}$ counting how often word $j$ appears in the context of word $i$ within the window. If we treat both left and right context symmetrically (which is the standard choice), this matrix is symmetric: $X_{ij} = X_{ji}$ .

Co-Occurrence Matrix

For a vocabulary of size $V$ , the co-occurrence matrix $\mathbf{X} \in \mathbb{R}^{V \times V}$ has entries $X_{ij}$ equal to the number of times word $j$ appears within a context window of word $i$ across the entire corpus. Larger windows capture broader, topical relationships; smaller windows capture tighter, syntactic relationships.

The window size matters more than it might seem. With a narrow window of size 2, a word's context consists of its immediate neighbors, so syntactic roles and local collocation patterns dominate. "Run" and "runs" will be close, and "fast" and "quickly" will be similar because they modify similar verbs. With a wide window of size 10, a word's context includes anything in the surrounding sentence or even beyond, so topically related words cluster together even if they never appear immediately adjacent. "Doctor" and "hospital" will be similar because they share many sentence-level co-occurrence partners.

From the raw counts we can compute conditional probabilities. Let $X_i = \sum_k X_{ik}$ be the total count of all words appearing in the context of word $i$ . Then:

P(j \mid i) = \frac{X_{ij}}{X_i}

where:

$X_{ij}$ : the number of times word $j$ appears in the context of word $i$
$X_i = \sum_k X_{ik}$ : the total number of context word occurrences for word $i$ (the row sum)

This is the probability of word $j$ appearing in the context of word $i$ . It normalizes the raw count by the total context budget for word $i$ , so frequent words like "the" do not automatically dominate.

The Ratio InsightLink Copied

The core motivation for GloVe comes from observing that ratios of conditional probabilities are more informative than the probabilities themselves. Consider two words: "ice" ( $i$ ) and "steam" ( $j$ ). We look at how each co-occurs with a probe word $k$ :

Co-occurrence probabilities and ratios for probe words relative to "ice" and "steam". Data from Pennington et al. (2014).
Probe word $k$	$P(k \mid \text{ice})$	$P(k \mid \text{steam})$	Ratio
solid	0.00019	0.000022	8.9
gas	0.000066	0.00078	0.085
water	0.003	0.0022	1.36
fashion	0.000017	0.000018	0.96

The ratios tell a far cleaner story than the raw probabilities:

"solid" has a high ratio: it is related to ice but not steam
"gas" has a low ratio: it is related to steam but not ice
"water" has a ratio near 1: it is related to both equally
"fashion" has a ratio near 1: it is related to neither

A good word vector model should encode this structure. The ratio $P(k|i)/P(k|j)$ discriminates relevant probe words from irrelevant ones far better than either probability alone.

Why are ratios so much better than raw probabilities? Consider what raw probabilities tell you. $P(\text{solid} | \text{ice}) = 0.00019$ is a tiny number that, taken alone, gives almost no information: it just says "solid" is somewhat rare in any context. But the ratio 8.9 immediately reveals a relationship: solid is about 9 times more associated with ice than with steam. The ratio normalizes away the baseline frequency of the probe word, leaving only the relative discriminative power.

This insight motivates GloVe's entire design. The model is built to encode these ratios in the geometry of the embedding space. When the vectors for "ice" and "steam" are represented as points in a high-dimensional space, their relative positions should reflect the pattern you see in the table above: "solid" should be much closer to "ice" than to "steam", "gas" should be the reverse, and "water" should sit equidistant.

In[4]:

Code

# Data from Pennington et al. (2014) GloVe paper
probe_words_table = ["solid", "gas", "water", "fashion"]
prob_ice = [1.9e-4, 6.6e-5, 3.0e-3, 1.7e-5]
prob_steam = [2.2e-5, 7.8e-4, 2.2e-3, 1.8e-5]
ratios = [p_i / p_s for p_i, p_s in zip(prob_ice, prob_steam)]

Out[5]:

Visualization

Bar chart of co-occurrence ratios for probe words relative to ice vs steam, showing high ratio for solid, low for gas, near-1 for water and fashion. — Co-occurrence probability ratios P(k|ice)/P(k|steam) for four probe words, shown on a log scale. Blue bars indicate probe words more associated with ice (ratio > 1), red bars indicate words more associated with steam (ratio < 1), and gray bars indicate neutral words (ratio near 1). The contrast between solid (8.64x) and gas (0.08x) demonstrates how ratios cleanly separate ice-specific from steam-specific concepts, while water and fashion remain near 1. GloVe's objective encodes these ratios directly in the word vector geometry.

The log scale makes the contrast clear. "Solid" has a ratio of about 8.9 (strongly associated with ice), "gas" has a ratio of about 0.085 (strongly associated with steam), while "water" and "fashion" cluster near 1. Ratios near 1 mean the probe word does not discriminate between ice and steam.