Pointwise Mutual Information › The Problem with Raw Counts

Before deriving PMI formally, it helps to see the problem with raw counts directly. Imagine a corpus where the word "bank" appears 500 times near "money" and 300 times near "river." At first glance, you might conclude that "bank" is more associated with financial contexts than geographic ones. But if "money" appears 50,000 times in the corpus and "river" appears only 1,000 times, then the expected co-occurrence with "bank" is much higher for "money" simply because "money" is far more common. The raw counts are misleading because they do not account for each word's overall prevalence.

This frequency-contamination problem is pervasive. In a typical English corpus, the ten most frequent words account for roughly 20-25% of all tokens. Words like "the," "of," "and," "in," and "to" appear so frequently that they will have large raw co-occurrence counts with virtually every content word in the vocabulary. If you were to rank context words for "Einstein" by raw co-occurrence count, "the" would sit near the top, which is not a meaningful finding. The same pattern would hold for "avocado," "chromosome," or any other content word, making raw counts nearly useless for identifying distinctive associations.

A second problem is that raw counts do not scale across corpora of different sizes. A co-occurrence count of 200 is significant evidence in a corpus of 10,000 words, but weak evidence in a corpus of 10 million words. When you compare results across corpora or combine data from different sources, raw counts become incomparable. You need a normalized measure that captures the same linguistic relationship regardless of corpus size.

The fix is to normalize by what we would expect given the individual frequencies of each word. If two words appear together exactly as often as chance predicts, PMI is zero. If they appear together more than chance predicts, PMI is positive. If they avoid each other, PMI is negative.

This normalization is the core idea behind PMI, and it converts co-occurrence data from a simple frequency table into a meaningful measure of linguistic association. The normalization is also the reason PMI correctly identifies that "Einstein" has a specific association with "relativity" even though the raw count may be modest, while "the" has no meaningful association with "Einstein" even though the raw count may be large.

The deeper problem with raw counts is that they conflate two distinct phenomena: the inherent commonness of individual words and the tendency of certain words to occur together. A word pair might have a high raw co-occurrence count for either reason, or both, and raw counts give you no way to distinguish.

Consider the word pair ("bank," "the"). In any large corpus, "the" will appear near "bank" thousands of times. But "the" also appears near every other noun with comparable frequency, because English grammar nearly requires it. The high co-occurrence count for ("bank," "the") tells you nothing about a special relationship between these words.

Now consider the pair ("bank," "deposit"). This pair might have a smaller absolute count, but if "deposit" appears near "bank" far more often than you would predict from its individual frequency, that excess is meaningful. "Deposit" has a specific semantic relationship with "bank" that "the" lacks. PMI measures this semantic signal by stripping out the background frequency contribution of each word.