Pointwise Mutual Information › Pointwise Mutual Information

Raw co-occurrence counts give you a starting point for measuring word associations, but they have a significant problem: common words pollute the signal. If "the" appears millions of times in your corpus, it will show high co-occurrence with nearly every other word, not because the relationships are meaningful, but simply because "the" is everywhere. The challenge is to separate specific association from accidental proximity.

Pointwise Mutual Information, or PMI, solves this problem by asking a more precise question: does word $w$ appear with context word $c$ more often than we would expect by chance? Instead of measuring raw counts, PMI measures the ratio between what we observe and what we would predict if the two words were statistically independent. A high PMI score means the two words co-occur far more than chance would predict, which is strong evidence of a real linguistic relationship. A low or negative PMI means they avoid each other or appear together only as often as expected by chance.

The elegance of PMI lies in how it converts a simple frequency table into a statement about statistical surprise. You can think of it as asking: "Given that I just saw word $w$, how much does that update my expectation of also seeing word $c$ nearby?" When the answer is "a lot," PMI is high. When seeing $w$ tells you nothing new about whether $c$ appears, PMI is zero. This framing, borrowed directly from information theory, gives PMI a principled foundation that ad hoc frequency cutoffs lack.

This chapter covers how PMI is derived from first principles, why it captures specific associations better than counts alone, and how variants like Positive PMI (PPMI) address PMI's practical shortcomings. We also implement PMI from scratch and explore its use for collocation extraction, a classical NLP task that remains relevant in modern NLP pipelines.