Pointwise Mutual Information › PMI Matrix Properties

Understanding the structural properties of the PPMI matrix helps you use it effectively and reason about its behavior before writing any code.

The raw PMI formula is symmetric: $\text{PMI}(w, c) = \text{PMI}(c, w)$. This follows directly from the formula, since swapping $w$ and $c$ leaves the joint probability unchanged and the product of marginals commutes. The symmetry also has an intuitive interpretation: if seeing "bank" increases your expectation of seeing "deposit," then seeing "deposit" increases your expectation of seeing "bank" by the same amount, assuming the same window-based co-occurrence definition.

In practice, many implementations construct asymmetric PMI matrices by distinguishing between target words (rows) and context words (columns). This directional setup can capture finer-grained relationships, such as tracking which words appear specifically to the left versus right of a target. Syntactic parsers use this asymmetry to model subject-verb and verb-object dependencies. For most distributional semantics tasks where the goal is measuring general semantic similarity, the symmetric version works well.

When the window is not symmetric around the target word, or when the co-occurrence counts are weighted by position, the resulting matrix may not be exactly symmetric even though the underlying PMI formula is. Always document which convention you use when comparing systems.

Even after applying PPMI, the matrix remains highly sparse. Most word pairs never co-occur within any window, so most entries are zero. In a vocabulary of 50,000 words, even a large corpus might fill only a small fraction of the $50{,}000 \times 50{,}000 = 2.5$ billion possible entries. Sparse matrix formats (such as CSR or COO in SciPy) are essential for handling PPMI matrices at scale, since storing the full dense matrix would require terabytes of memory.

Sparsity is not purely a computational burden. The zero entries encode an observed absence of association, and preserving that structure is important for downstream tasks like cosine similarity computation. When you compare two PPMI vectors, the dimensions where both are zero do not contribute to the cosine similarity, which is the correct behavior: shared absence of association is not evidence of semantic similarity.

The sparsity pattern also reveals something about the vocabulary. Function words have dense rows in the raw co-occurrence matrix (they appear near everything), but after PPMI their rows become sparser because their high individual frequency makes co-occurrences unsurprising. Rare technical terms may have very sparse PPMI rows because they appear near only a handful of other terms. This reflects the underlying semantic specificity: specialized vocabulary has fewer but more specific associations.

Each row of the PPMI matrix is a vector representation of the corresponding word, where each dimension corresponds to a context word and the value indicates the strength of association. Words that share similar context distributions will have similar PPMI row vectors, measurable by cosine similarity. This is the direct computational realization of the distributional hypothesis: words used in similar contexts receive similar PPMI vectors.

Consider the words "cat" and "dog." In any reasonably large corpus, they will appear near many of the same context words: "pet," "owner," "feed," "animal," "veterinarian." Their PPMI vectors will have high values at many of the same dimensions, producing a high cosine similarity. The word "rock," by contrast, will share fewer high-PPMI contexts with "cat," producing a lower similarity score.

These raw PPMI vectors can be used directly for word similarity tasks, but they are high-dimensional and sparse. Applying Singular Value Decomposition to the PPMI matrix, which we cover in the next chapter on LSA, compresses this information into dense, low-dimensional vectors that often capture semantic relationships even more cleanly. The SVD finds the most important patterns in the PPMI matrix and discards the noise, giving you a compact representation that generalizes better to unseen word pairs.

The distributional hypothesis, formalized by linguist John Rupert Firth in the 1950s with the memorable phrase "a word is characterized by the company it keeps," gives the theory behind all co-occurrence-based methods including PMI. The hypothesis says that semantic similarity can be inferred from distributional similarity: if two words appear in similar contexts across a large corpus, they are likely to be semantically related.

PMI operationalizes this hypothesis at the level of individual word-context pairs. Rather than comparing entire context distributions directly (which raw co-occurrence counts attempt but fail to do cleanly), PMI asks which specific contexts are distinctively associated with each word. Words with high PMI associations to similar distinctive contexts will end up as similar vectors.

This is why PMI works better than raw counts for semantic tasks: it identifies what is distinctive about a word's usage, not just what is frequent around it. "Coffee" and "espresso" both appear frequently near "hot" and "drink," but they also both have high PMI with "cafe," "roast," and "grind" in a way that "hot chocolate" does not share, even though all three are hot drinks. PMI can capture this more fine-grained semantic structure.