Pointwise Mutual Information: Word Associations in NLP

Raw co-occurrence counts give you a starting point for measuring word associations, but they have a significant problem: common words pollute the signal. If "the" appears millions of times in your corpus, it will show high co-occurrence with nearly every other word, not because the relationships are meaningful, but simply because "the" is everywhere. The challenge is to separate specific association from accidental proximity.

Pointwise Mutual Information, or PMI, solves this problem by asking a more precise question: does word $w$ appear with context word $c$ more often than we would expect by chance? Instead of measuring raw counts, PMI measures the ratio between what we observe and what we would predict if the two words were statistically independent. A high PMI score means the two words co-occur far more than chance would predict, which is strong evidence of a real linguistic relationship. A low or negative PMI means they avoid each other or appear together only as often as expected by chance.

The elegance of PMI lies in how it converts a simple frequency table into a statement about statistical surprise. You can think of it as asking: "Given that I just saw word $w$, how much does that update my expectation of also seeing word $c$ nearby?" When the answer is "a lot," PMI is high. When seeing $w$ tells you nothing new about whether $c$ appears, PMI is zero. This framing, borrowed directly from information theory, gives PMI a principled foundation that ad hoc frequency cutoffs lack.

This chapter covers how PMI is derived from first principles, why it captures specific associations better than counts alone, and how variants like Positive PMI (PPMI) address PMI's practical shortcomings. We also implement PMI from scratch and explore its use for collocation extraction, a classical NLP task that remains relevant in modern NLP pipelines.

Before deriving PMI formally, it helps to see the problem with raw counts directly. Imagine a corpus where the word "bank" appears 500 times near "money" and 300 times near "river." At first glance, you might conclude that "bank" is more associated with financial contexts than geographic ones. But if "money" appears 50,000 times in the corpus and "river" appears only 1,000 times, then the expected co-occurrence with "bank" is much higher for "money" simply because "money" is far more common. The raw counts are misleading because they do not account for each word's overall prevalence.

This frequency-contamination problem is pervasive. In a typical English corpus, the ten most frequent words account for roughly 20-25% of all tokens. Words like "the," "of," "and," "in," and "to" appear so frequently that they will have large raw co-occurrence counts with virtually every content word in the vocabulary. If you were to rank context words for "Einstein" by raw co-occurrence count, "the" would sit near the top, which is not a meaningful finding. The same pattern would hold for "avocado," "chromosome," or any other content word, making raw counts nearly useless for identifying distinctive associations.

A second problem is that raw counts do not scale across corpora of different sizes. A co-occurrence count of 200 is significant evidence in a corpus of 10,000 words, but weak evidence in a corpus of 10 million words. When you compare results across corpora or combine data from different sources, raw counts become incomparable. You need a normalized measure that captures the same linguistic relationship regardless of corpus size.

The fix is to normalize by what we would expect given the individual frequencies of each word. If two words appear together exactly as often as chance predicts, PMI is zero. If they appear together more than chance predicts, PMI is positive. If they avoid each other, PMI is negative.

This normalization is the core idea behind PMI, and it converts co-occurrence data from a simple frequency table into a meaningful measure of linguistic association. The normalization is also the reason PMI correctly identifies that "Einstein" has a specific association with "relativity" even though the raw count may be modest, while "the" has no meaningful association with "Einstein" even though the raw count may be large.

The deeper problem with raw counts is that they conflate two distinct phenomena: the inherent commonness of individual words and the tendency of certain words to occur together. A word pair might have a high raw co-occurrence count for either reason, or both, and raw counts give you no way to distinguish.

Consider the word pair ("bank," "the"). In any large corpus, "the" will appear near "bank" thousands of times. But "the" also appears near every other noun with comparable frequency, because English grammar nearly requires it. The high co-occurrence count for ("bank," "the") tells you nothing about a special relationship between these words.

Now consider the pair ("bank," "deposit"). This pair might have a smaller absolute count, but if "deposit" appears near "bank" far more often than you would predict from its individual frequency, that excess is meaningful. "Deposit" has a specific semantic relationship with "bank" that "the" lacks. PMI measures this semantic signal by stripping out the background frequency contribution of each word.

PMI comes from information theory, specifically from the concept of mutual information between random variables. Mutual information asks how much knowing one variable tells you about another. It quantifies the reduction in uncertainty about one variable when you know the value of another. Pointwise mutual information applies this question to a specific pair of outcomes rather than to entire distributions.

Think of each word occurrence as a random event. When you sample a position in the corpus at random, you draw a word. Define:

• $P(w)$: the probability of randomly sampling word $w$ from the corpus
• $P(c)$: the probability of randomly sampling context word $c$
• $P(w, c)$: the probability of observing the word pair $(w, c)$ together in a co-occurrence event

If $w$ and $c$ were completely independent, we would expect their joint probability to equal the product of their marginal probabilities:

This is simply the definition of statistical independence. Two events are independent if and only if their joint probability factors into the product of the marginals. If you know that "bank" and "deposit" are independent, then knowing "bank" appeared tells you nothing about whether "deposit" appeared nearby.

PMI measures the log-ratio of the observed joint probability to this independent baseline:

where:

• $P(w, c)$: the observed probability of word $w$ appearing with context $c$
• $P(w)$: the marginal probability of word $w$ across all contexts
• $P(c)$: the marginal probability of context $c$ across all co-occurrences
• $\log_2$: logarithm base 2, so PMI values are expressed in bits

When the observed probability equals the expected probability under independence, the ratio is 1 and $\log_2(1) = 0$. When the words co-occur more than chance predicts, the ratio exceeds 1 and PMI is positive. When words co-occur less than chance predicts, the ratio falls below 1 and PMI is negative.

The logarithm serves several important roles. First, it converts the ratio from a multiplicative scale to an additive one, which is easier to work with mathematically and algorithmically. Second, it gives PMI a natural interpretation in information theory: a PMI of 1 bit means that observing $w$ doubles your probability estimate for $c$. Third, the logarithm makes PMI symmetric in the sense that $\text{PMI}(w, c) = \text{PMI}(c, w)$, which you can verify algebraically since the joint probability and the product of marginals are both symmetric.

In practice, we estimate these probabilities from a co-occurrence matrix. Given a corpus with $N$ total co-occurrence events:

• $P(w) = \dfrac{\text{count}(w)}{N}$, where $\text{count}(w)$ is the total number of co-occurrence events involving word $w$
• $P(c) = \dfrac{\text{count}(c)}{N}$, where $\text{count}(c)$ is the total number of co-occurrence events involving context $c$
• $P(w, c) = \dfrac{\text{count}(w, c)}{N}$, where $\text{count}(w, c)$ is the number of times $w$ and $c$ co-occur within the context window

Substituting these into the PMI formula and simplifying:

This final form is what you implement in code: multiply the raw co-occurrence count by the total event count $N$, then divide by the product of the two marginal counts. The logarithm converts this ratio into an additive scale where zero means independence. Notice that $N$ cancels from the marginal probabilities but remains in the joint probability, which is why it appears in the numerator of the count-based formula. The ratio $\text{count}(w,c) \cdot N$ in the numerator can be understood as the observed count scaled up to match the units of the denominator $\text{count}(w) \cdot \text{count}(c)$.

PMI values have clear, intuitive meanings that map directly onto the concept of statistical association:

• PMI = 0: The words appear together exactly as often as chance predicts. No evidence of association beyond their individual frequencies.
• PMI > 0: The words appear together more than chance predicts. Higher values mean stronger, more specific association. A PMI of 2 bits means the pair co-occurs four times more often than chance.
• PMI < 0: The words appear together less than chance predicts. They tend to avoid each other, though this is often an artifact of data sparsity rather than actual repulsion.
• PMI approaching $+\infty$: Occurs when two words almost always appear together but one or both are very rare. This can happen with highly specialized terminology or named entities in small corpora.
• PMI $= -\infty$: Occurs when $\text{count}(w, c) = 0$. The words never co-occur, making the log of zero undefined.

The $-\infty$ case is a serious practical problem. Many word pairs never co-occur, especially in large vocabularies with limited data. A vocabulary of 50,000 words yields a matrix with 2.5 billion cells, but even a large corpus might only fill a small fraction of them. This leads directly to the concept of Positive PMI.

A caveat is that PMI values for rare words can be very high even when based on flimsy evidence. If two words each appear only twice in the corpus but always together, their PMI is very high. That sounds like strong evidence, but two observations are not enough to establish an association. This statistical fragility for low-frequency pairs is one of PMI's main practical weaknesses and motivates the smoothing and thresholding strategies discussed later.

PMI is grounded in mutual information rather than being an ad hoc normalization. It has a direct connection to mutual information, one of the foundational quantities in information theory. Mutual information between two random variables $W$ and $C$ is defined as:

This is exactly a weighted average of PMI values, where the weights are the joint probabilities $P(w, c)$. Mutual information measures the total statistical dependency between $W$ and $C$ across all possible word and context pairs. Pointwise mutual information zooms in on a single pair $(w, c)$ and reports the contribution of that specific combination to the overall mutual information.

This connection matters for two reasons. First, it means PMI has a well-founded theoretical interpretation, rather than a purely heuristic one. Second, it means that maximizing PMI for specific pairs is related to, though not identical to, maximizing mutual information across all pairs, which is the objective of many representation learning algorithms.

The most widely used variant of PMI is Positive PMI (PPMI), which replaces all negative values (including $-\infty$) with zero:

where:

• $\text{PMI}(w, c)$: the pointwise mutual information as computed above
• $\max(0, \cdot)$: clips the value at zero, replacing any negative or undefined result with 0

This single modification handles the zero-count problem cleanly: any word pair that never co-occurs has $\text{PMI} = -\infty$, and PPMI converts that to 0. PPMI also discards actual negative associations, treating "never seen together" and "seen together less often than chance" both as zero. This is a deliberate simplification: for most NLP tasks, we care about positive associations. A high PPMI score reliably indicates a real linguistic relationship, while negative scores are harder to interpret reliably, particularly because many negative values arise from data sparsity rather than systematic avoidance.

PPMI also has a valuable scale-invariance property. Doubling the corpus size does not change PPMI scores, because both the joint and marginal probabilities scale proportionally. Raw co-occurrence counts, by contrast, simply grow with corpus size, making cross-corpus comparisons unreliable.

There is also a practical engineering benefit to PPMI: it produces non-negative vectors. Many downstream applications, including certain dimensionality reduction algorithms and cosine similarity computations, benefit from or require non-negative inputs. A PPMI vector is a legitimate sparse non-negative vector where each entry represents the strength of a specific word-context association.

The decision to replace negative PMI values with zero deserves more explanation, because it might seem like we are discarding information. The key insight is that negative PMI values are epistemically much weaker than positive ones.

A positive PMI value says: "I observed this pair appearing together more often than chance would predict." That is a positive empirical signal. The higher the count, the more confident you can be.

A negative PMI value, in the range available from finite corpora, says only: "I did not observe this pair appearing together as often as chance would predict." But absence of evidence is weak evidence of absence, especially when the corpus is finite. If "bank" and "octopus" have never appeared in the same window in your corpus, that might be because they systematically avoid each other, or it might be because your corpus simply doesn't contain enough diverse text to cover every conceivable word combination. In a sufficiently large corpus, almost every word appears near almost every other word at least once, so the zero-count entries are an artifact of corpus size, not a meaningful linguistic fact.

This asymmetry in reliability is why PPMI is standard practice. Positive values are trustworthy signals; negative and zero values are uninformative.

PMI has a well-known bias toward rare words. If two words appear together only once but each word is itself rare in the corpus, their PMI can be extremely high even though the evidence from a single observation is weak and statistically unreliable. One co-occurrence of a rare pair is not enough to establish an association.

Conversely, very frequent words like "the" and "of" tend to have large absolute co-occurrence counts but low PMI, because their high individual frequencies make high co-occurrence expected. PMI correctly identifies that "the" has no special relationship with "bank" despite often appearing near it.

The result is that PMI rewards rare, specific word pairs and penalizes common words. This is usually what we want: we care more that "asylum" and "seeker" associate strongly than that "the" and "is" associate. But the rare-word bias can cause problems when the vocabulary contains many low-frequency terms with minimal evidence. A word that appears three times in the corpus, all three times near the same context word, will receive a very high PMI for that context, but the signal is statistically fragile.

In practice, imposing a minimum co-occurrence count threshold (discarding pairs with fewer than five or ten co-occurrences) substantially reduces this noise. More principled approaches include Laplace smoothing (adding a small pseudocount to all matrix cells before computing probabilities), discounting methods borrowed from language modeling, or using the shifted PPMI variant described next.

One principled fix for the rare-word bias is shifted PPMI (SPPMI), introduced by Levy and Goldberg (2014) in their influential analysis of word2vec. They showed that the skip-gram model with negative sampling implicitly factorizes a shifted PMI matrix. The shifted variant is:

where:

• $k$: the number of negative samples, a hyperparameter matching word2vec's negative sampling count
• $\log_2 k$: the constant shift subtracted from each PMI value before clipping

With $k > 1$, the shift increases the threshold a word pair must exceed before contributing a nonzero value to the matrix. This effectively filters out weakly associated pairs and reduces the influence of rare co-occurrences that inflate standard PPMI. Common choices for $k$ are 1, 5, and 15, corresponding to typical negative sampling values used in word2vec training.

To understand why this shift makes sense, recall that word2vec with negative sampling trains a model to distinguish real context words from randomly sampled "noise" words. With $k$ noise samples per real context, the model learns to assign high scores to pairs that appear $k$ times more often than would be expected from random sampling. This is precisely the condition that $\text{PMI}(w,c) > \log_2 k$ captures. The shift aligns the PPMI threshold with the implicit threshold used by the neural model.

The connection Levy and Goldberg uncovered is theoretically significant: it means that training a neural skip-gram model is, in effect, doing matrix factorization on a shifted PMI matrix. The two approaches, one classical and statistical, one neural and gradient-based, are optimizing related objectives. This unification demonstrated that decades of distributional semantics research and modern neural word embeddings were more closely related than practitioners had assumed. The neural model's apparent advantage was not that it discovered fundamentally different structure in language, but that it used a different, often more effective, factorization method (stochastic gradient descent on the implicit matrix) combined with architectural choices like context smoothing and subsampling that are not part of the basic PPMI formulation.

Understanding the structural properties of the PPMI matrix helps you use it effectively and reason about its behavior before writing any code.

The raw PMI formula is symmetric: $\text{PMI}(w, c) = \text{PMI}(c, w)$. This follows directly from the formula, since swapping $w$ and $c$ leaves the joint probability unchanged and the product of marginals commutes. The symmetry also has an intuitive interpretation: if seeing "bank" increases your expectation of seeing "deposit," then seeing "deposit" increases your expectation of seeing "bank" by the same amount, assuming the same window-based co-occurrence definition.

In practice, many implementations construct asymmetric PMI matrices by distinguishing between target words (rows) and context words (columns). This directional setup can capture finer-grained relationships, such as tracking which words appear specifically to the left versus right of a target. Syntactic parsers use this asymmetry to model subject-verb and verb-object dependencies. For most distributional semantics tasks where the goal is measuring general semantic similarity, the symmetric version works well.

When the window is not symmetric around the target word, or when the co-occurrence counts are weighted by position, the resulting matrix may not be exactly symmetric even though the underlying PMI formula is. Always document which convention you use when comparing systems.

Even after applying PPMI, the matrix remains highly sparse. Most word pairs never co-occur within any window, so most entries are zero. In a vocabulary of 50,000 words, even a large corpus might fill only a small fraction of the $50{,}000 \times 50{,}000 = 2.5$ billion possible entries. Sparse matrix formats (such as CSR or COO in SciPy) are essential for handling PPMI matrices at scale, since storing the full dense matrix would require terabytes of memory.

Sparsity is not purely a computational burden. The zero entries encode an observed absence of association, and preserving that structure is important for downstream tasks like cosine similarity computation. When you compare two PPMI vectors, the dimensions where both are zero do not contribute to the cosine similarity, which is the correct behavior: shared absence of association is not evidence of semantic similarity.

The sparsity pattern also reveals something about the vocabulary. Function words have dense rows in the raw co-occurrence matrix (they appear near everything), but after PPMI their rows become sparser because their high individual frequency makes co-occurrences unsurprising. Rare technical terms may have very sparse PPMI rows because they appear near only a handful of other terms. This reflects the underlying semantic specificity: specialized vocabulary has fewer but more specific associations.

Each row of the PPMI matrix is a vector representation of the corresponding word, where each dimension corresponds to a context word and the value indicates the strength of association. Words that share similar context distributions will have similar PPMI row vectors, measurable by cosine similarity. This is the direct computational realization of the distributional hypothesis: words used in similar contexts receive similar PPMI vectors.

Consider the words "cat" and "dog." In any reasonably large corpus, they will appear near many of the same context words: "pet," "owner," "feed," "animal," "veterinarian." Their PPMI vectors will have high values at many of the same dimensions, producing a high cosine similarity. The word "rock," by contrast, will share fewer high-PPMI contexts with "cat," producing a lower similarity score.

These raw PPMI vectors can be used directly for word similarity tasks, but they are high-dimensional and sparse. Applying Singular Value Decomposition to the PPMI matrix, which we cover in the next chapter on LSA, compresses this information into dense, low-dimensional vectors that often capture semantic relationships even more cleanly. The SVD finds the most important patterns in the PPMI matrix and discards the noise, giving you a compact representation that generalizes better to unseen word pairs.

The distributional hypothesis, formalized by linguist John Rupert Firth in the 1950s with the memorable phrase "a word is characterized by the company it keeps," gives the theory behind all co-occurrence-based methods including PMI. The hypothesis says that semantic similarity can be inferred from distributional similarity: if two words appear in similar contexts across a large corpus, they are likely to be semantically related.

PMI operationalizes this hypothesis at the level of individual word-context pairs. Rather than comparing entire context distributions directly (which raw co-occurrence counts attempt but fail to do cleanly), PMI asks which specific contexts are distinctively associated with each word. Words with high PMI associations to similar distinctive contexts will end up as similar vectors.

This is why PMI works better than raw counts for semantic tasks: it identifies what is distinctive about a word's usage, not just what is frequent around it. "Coffee" and "espresso" both appear frequently near "hot" and "drink," but they also both have high PMI with "cafe," "roast," and "grind" in a way that "hot chocolate" does not share, even though all three are hot drinks. PMI can capture this more fine-grained semantic structure.

Let's work through a small example to build intuition. Consider a tiny corpus of four sentences:

"The cat chases mice." "The dog chases cats." "Cats and dogs are common pets." "Mice fear cats."

Using a window size of 1 (immediate neighbors only), we count co-occurrences and compute PMI for the pair (cat, chases).

Suppose after counting we have:

• $\text{count}(\text{cat}, \text{chases}) = 2$
• $\text{count}(\text{cat}) = 4$ (total number of co-occurrence events in which "cat" participates)
• $\text{count}(\text{chases}) = 3$ (total number of co-occurrence events in which "chases" participates)
• $N = 20$ (total co-occurrence events across all word pairs)

The estimated probabilities are:

The expected joint probability under the independence assumption would be:

Computing PMI:

A PMI of 1.74 bits means "cat" and "chases" co-occur about 3.3 times more often than chance would predict. This represents a meaningful association: the corpus provides evidence that cats chase things.

Now compare this with the pair (the, chases). Suppose "the" appears so frequently that $P(\text{the}) = 0.25$, but co-occurs with "chases" only about as often as expected by chance. Under near-independence, the ratio is close to 1 and:

PMI correctly reveals that "the" and "chases" have no special relationship, even though "the" may appear many times near "chases" in absolute terms.

Let's also verify the count-based formula directly to make sure the algebra is clear. From the derivation above:

Plugging in the values from the (cat, chases) example:

The count-based formula gives the same result as the probability-based formula, as expected. In code, you will typically use the count-based form because it avoids explicitly computing probabilities and works directly with the co-occurrence matrix.

Now suppose we also compute PMI for a pair with only one co-occurrence, where both words are relatively rare. Say $\text{count}(w_1, w_2) = 1$, $\text{count}(w_1) = 2$, $\text{count}(w_2) = 2$, and $N = 20$:

This PMI of 2.32 is higher than the (cat, chases) PMI of 1.74, even though the evidence is much thinner: a single co-occurrence vs. two. This illustrates the rare-word bias. PPMI retains this value as-is, which is why minimum count thresholds are important in practice.

And for a pair that never co-occurs, $\text{count}(w, c) = 0$:

PPMI converts this to 0, handling the undefined case cleanly.

Let's implement PMI from scratch, then visualize the differences between raw counts and PPMI to make the normalization effect concrete.

We will build everything from a small but realistic text corpus with sentences from multiple semantic domains. Using multiple domains lets us see whether PMI correctly separates the financial meaning of "bank" from its geographic meaning, and whether it correctly identifies that "cat" and "dog" share more contexts with each other than with words from the financial domain.

Our small corpus has a manageable vocabulary, which lets us inspect the full co-occurrence and PPMI matrices directly. The two-domain structure (animals vs. banking/rivers) will let us verify that PMI correctly identifies cross-domain words like "bank" as having associations in both camps.

We construct the raw co-occurrence matrix using a sliding window, as covered in the previous chapter on co-occurrence matrices. Each sentence contributes co-occurrence counts for all pairs of words within the window around each target word.

The high sparsity is typical: most word pairs never appear within a window of each other, so most matrix entries are zero. Even with a window size of 2, only a small fraction of all possible word pairs appear as neighbors in these sentences.

Now we compute the PPMI matrix from the raw co-occurrence counts. The key steps are computing the marginal and joint probabilities, forming the PMI ratio, and clipping at zero. The implementation supports both standard PPMI (with $k=1$) and shifted PPMI (with $k > 1$), allowing easy experimentation with the shift parameter.

The np.where call handles two tricky cases simultaneously. When cooc_matrix > 0 but denominator == 0, we have a marginal probability of zero, which should not occur in a well-formed matrix but can arise if a word has been zero-padded. When cooc_matrix == 0, the PMI is $-\infty$ regardless of the denominator, so we assign $-\infty$ directly. The final np.maximum(0, pmi) then clips both $-\infty$ and any finite negative values to zero, producing the PPMI matrix.

With the PPMI matrix built, we can look up specific word associations. Let's examine the top associations for a few target words and compare the PPMI scores against raw co-occurrence counts.

The PPMI scores reveal specific semantic relationships. "Cat" associates strongly with words like "chased," "mouse," and "mat," reflecting actual usage in the corpus. The word "bank" reveals both its financial and geographic contexts through different high-PMI context words. High-frequency function words like "the" and "and" get suppressed: even if "the" appears many times near "cat" in absolute terms, PMI recognizes that "the" appears near everything equally, so the association is unremarkable.

The side-by-side display of PPMI and raw counts makes this suppression concrete. A context word with a high raw count but low PPMI is a function word or very common noun. A context word with a high PPMI is semantically distinctive to the target word.

Let's directly compare rankings produced by raw counts versus PPMI. This makes the normalization effect concrete and shows exactly which words get promoted and which get demoted when you switch from raw frequency to association strength.

Notice how raw counts favor high-frequency words that appear near everything, while PPMI promotes words with a specific association to the target word. This shift in ranking is the core benefit of PMI: you see the words that are truly distinctive to "cat" rather than simply common in the corpus.

Once we have PPMI vectors, we compute word similarity using cosine similarity. Two words are similar if their PPMI context distributions are aligned: they share the same high-association contexts, indicating they are used in similar linguistic environments.

Cosine similarity between two vectors $\mathbf{u}$ and $\mathbf{v}$ is defined as:

where:

• $\mathbf{u} \cdot \mathbf{v}$: the dot product, summing the products of corresponding dimensions
• $\|\mathbf{u}\|$: the Euclidean norm of $\mathbf{u}$
• $\|\mathbf{v}\|$: the Euclidean norm of $\mathbf{v}$

Cosine similarity ranges from $-1$ to $+1$, though with PPMI vectors (which are non-negative), it ranges from $0$ to $+1$. A value of 1 means the two vectors point in exactly the same direction, indicating identical context distributions. A value of 0 means the vectors are orthogonal, indicating no shared high-association contexts.

"Cat" and "dog" share many contexts (both are common animals that interact with the same set of words in the corpus), which produces a high similarity score. "Cat" and "bank" share few meaningful contexts, producing a low similarity score. "Bank" and "river" show moderate similarity. This reflects their shared geographic context in the corpus sentences where "river bank" occurs.

These results match human intuition about semantic similarity, and they emerge entirely from statistical patterns in the text without any hand-crafted features or curated knowledge bases.

One of the original applications of PMI is collocation extraction: finding word pairs that form meaningful multi-word expressions. Collocations are combinations whose meaning or usage cannot be predicted from their parts alone. "Hot dog," "kick the bucket," and "New York" all qualify as collocations. A statistical association measure is needed because simple frequency is insufficient: "a the" appears frequently but is not a collocation, while "kick the bucket" appears infrequently but is.

PMI is well-suited to this task because it rewards specificity. "Hot" has relatively high PMI with "dog" because the combination "hot dog" is more specific than you would expect from the individual frequencies of each word. Compare this to "hot water," where "hot" and "water" might have lower PMI because both are common and their combination, while frequent, is not particularly surprising given that hot liquids and water are common topics.

The highest-PMI pairs represent words that consistently appear together in the corpus. Even in this small dataset, PMI identifies consistent co-occurrence patterns rather than incidental overlaps driven by word frequency. In a production collocation detector, you would apply this to a much larger corpus and add minimum frequency thresholds to avoid high-PMI pairs that are based on only one or two co-occurrence events.

Visualizations make the effect of PMI normalization immediately clear in ways that tables cannot. The following plots examine the PPMI matrix structure, compare PMI against raw counts for a set of context words, and illustrate how different PMI shift values affect the resulting distribution.

A heatmap of the PPMI matrix for a focused subset of words reveals the association structure directly. By selecting words from different semantic domains, we can see whether PMI recovers the domain boundaries from raw text.

The heatmap clearly shows that words from the same semantic domain share high PPMI scores with each other. Animal-domain words ("cat," "dog," "mouse") show mutual associations, while financial words ("bank," "fees," "loan") form their own cluster. The word "bank" shows some crossover. This reflects its use in both financial and geographic contexts in the corpus.

Shifted PPMI with different values of $k$ changes the distribution of positive associations. Larger $k$ raises the threshold, retaining only the strongest associations and pruning pairs whose PMI exceeds independence only by a small margin.

Standard PPMI ($k=1$) retains all positive associations, including weakly associated pairs near zero. Shifted PPMI ($k=5$) discards these borderline pairs, retaining only word combinations with strong, reliable co-occurrence evidence. The trade-off is coverage versus precision: $k=1$ covers more pairs but includes more noise; larger $k$ is more conservative but more reliable.

In practice, choosing $k$ depends on the downstream task. For collocation extraction, where precision matters more than recall, a larger shift is appropriate. For constructing word vectors for a downstream similarity task, a smaller shift preserves more information about the distributional structure.

This visualization makes the ranking shift from raw counts to PPMI concrete for a set of target context words, confirming the analytical argument with an empirical comparison.

The contrast between the two rankings is striking. Raw counts prominently feature function words that appear near everything. PPMI suppresses these and surfaces words with direct semantic relevance to "cat," such as the animals it interacts with and the verbs that describe its behavior. This is the normalization effect that makes PPMI useful for downstream NLP tasks.

The basic PPMI formula works well for large, balanced corpora, but several practical issues arise in real deployments that require additional engineering decisions.

The simplest and most effective way to control the rare-word problem is to apply a minimum frequency threshold before computing PPMI. Any word that appears fewer than $k$ times in the corpus is excluded from the vocabulary, and any co-occurrence pair with fewer than $m$ joint occurrences is treated as zero. Typical values are $k = 5$ for vocabulary thresholds and $m = 2$ or $m = 5$ for co-occurrence thresholds.

This approach has several benefits. It reduces the vocabulary size, which reduces memory requirements for the co-occurrence matrix. It eliminates the highest-variance PMI estimates, those based on only one or two co-occurrence events. And it focuses the representation on the part of the vocabulary with sufficient evidence for reliable estimates.

The main cost is coverage: words that appear fewer than five times in the corpus are excluded entirely, which can be significant for technical domains with specialized vocabulary or for morphologically rich languages where every inflected form is counted separately.

Laplace smoothing (also called add-$\alpha$ smoothing) addresses the zero-count problem in a more principled way than simply discarding rare pairs. Instead of working with the raw counts $\text{count}(w, c)$, you add a small pseudocount $\alpha$ to every cell:

With $\alpha = 1$ (standard Laplace smoothing) or $\alpha < 1$ (Lidstone smoothing), every word pair gets a small baseline probability even if it was never observed. This prevents $-\infty$ from appearing in the PMI computation and reduces the variance of estimates for rare pairs.

The practical effect is that the PPMI matrix becomes denser: pairs that would have been zero now have small positive values. This can improve performance on downstream tasks that use the PPMI matrix as input to a learning algorithm, because the algorithm receives a more complete picture of the distributional structure.

Some PPMI implementations apply sublinear scaling to the co-occurrence counts before computing probabilities. One common variant counts each co-occurrence event with weight $1 / d$ where $d$ is the distance between the two words within the window. Words that are immediately adjacent receive full weight, while words at the far edge of the window receive reduced weight.

This position-weighted counting better reflects the intuition that nearby words are more informative context than distant ones. The resulting PPMI values place more weight on tight, adjacent associations and less on loose, distant ones. GloVe uses a related but distinct weighting function: the weight decreases with distance but is bounded by a maximum value of 1 to avoid over-weighting very frequent immediate neighbors.

Levy and Goldberg also noted that word2vec implicitly raises context word frequencies to the power $3/4$ before using them as negative sampling probabilities. This smoothing reduces the dominance of very high-frequency context words in the model's training signal. The equivalent for PPMI is to raise the context word counts to the power $\alpha$ (typically $\alpha = 0.75$) before computing column marginals:

where:

• $\alpha$: the smoothing exponent, typically 0.75
• $\text{count}(c)^\alpha$: the smoothed count for context word $c$

The modified PPMI using this smoothed marginal is:

This smoothing reduces the raw PMI of pairs involving very high-frequency context words, bringing the PPMI formulation closer to the implicit objective of word2vec with negative sampling. In practice, this variant consistently outperforms standard PPMI on word similarity benchmarks.

PMI is not limited to word-word co-occurrence. The same formula applies whenever you want to measure the association between any two categorical variables from count data, and this generality has made it useful across a wide range of NLP applications.

In information retrieval, PMI is used to weight word-document associations. Instead of TF-IDF (which combines term frequency with inverse document frequency), you can compute PMI between each word and each document:

where $P(w, d)$ is the probability of observing word $w$ in document $d$, $P(w)$ is the overall word frequency, and $P(d)$ is the document's relative length. High PMI for (word, document) pairs identifies words that are distinctive of specific documents, similar in spirit to TF-IDF but derived from a different theoretical foundation.

Named entity recognition and relation extraction systems have historically used PMI to identify features that are distinctive of entity classes. For example, you might compute PMI between each word and the label "PERSON" based on annotated training data. Words with high PMI for "PERSON" (like "Mr.," "Ms.," capitalized names) become strong features for the classifier, while words with PMI near zero are not discriminative for that class.

Turney (2002) used PMI to compute the semantic orientation of adjectives for sentiment analysis, in a method that predates the deep learning era but remains influential. The idea is to compute the PMI between each adjective and two reference words representing positive sentiment ("excellent") and negative sentiment ("poor"), then take the difference:

where:

• $\text{SO}(w)$: the semantic orientation score for word $w$
• $\text{PMI}(w, \text{excellent})$: how strongly $w$ associates with positive sentiment
• $\text{PMI}(w, \text{poor})$: how strongly $w$ associates with negative sentiment

An adjective with positive SO ("wonderful," "brilliant") co-occurs more strongly with "excellent" than with "poor." An adjective with negative SO ("terrible," "mediocre") has the reverse pattern. This simple approach achieved surprisingly strong performance on review classification tasks. This shows that PMI can capture meaningful semantic properties without any supervised training.

PMI has been enormously influential in NLP, but it comes with several well-understood limitations that practitioners need to keep in mind.

The most significant practical limitation is sensitivity to rare events. When a word pair appears only once or twice, the PMI estimate is unreliable. A pair might score very high simply because one or both words are rare in general, making any co-occurrence appear remarkable by chance. This is why PPMI benefits from additional smoothing or minimum count thresholds in production systems. The problem worsens with larger vocabularies: a vocabulary of 100,000 words produces a $100{,}000 \times 100{,}000$ matrix with 10 billion cells, the vast majority of which will be empty, and the non-zero cells will often have very small counts.

PMI also ignores word order within the context window. The pair (cat, chased) scores the same regardless of whether the cat is doing the chasing or being chased. For many tasks this is fine, but for understanding more detailed semantic relationships like agent-patient roles, you would need directional weighting or syntactic context. Similarly, all positions within the window contribute equally by default. A word appearing immediately adjacent to the target is treated the same as a word appearing at the window boundary, even though immediate neighbors are usually more informative. Position-weighted variants address this but add complexity.

A third limitation is the binary nature of co-occurrence windows. PMI measures whether two words appeared within a window, but it does not distinguish between a pair that appears ten times at distance 1 and a pair that appears ten times at distances 1 through 5. More sophisticated approaches, including GloVe and neural word embeddings, use weighted or richer context representations that capture more positional information.

Despite these limitations, PMI made several lasting contributions to NLP. It provided a principled, interpretable way to measure word association, replacing simple frequency thresholds and ad hoc weighting schemes. PPMI matrices served as the foundation for Latent Semantic Analysis (LSA) and its successors, proving that dense, low-dimensional word representations could capture meaningful semantic properties. When Levy and Goldberg showed that word2vec implicitly factorizes a shifted PMI matrix, it revealed that neural word embeddings and classical distributional models are mathematically related, unifying decades of research under a common framework.

For collocation extraction specifically, PMI remains competitive with more sophisticated methods. The association between a trigram like "New York City" or a bigram like "kick the bucket" is well-captured by PMI, which correctly identifies that these combinations appear far more often than chance would predict. This makes PMI a practical tool in applications ranging from phrase detection in search engines to terminology extraction in specialized domains.

PMI also remains widely used as a baseline and evaluation tool. When developing new word embedding techniques, comparing against PPMI vectors provides an interpretable reference point. Since PPMI vectors are sparse and directly interpretable (each dimension corresponds to a specific context word), they make it easy to audit what a model has learned and diagnose failure modes. If your neural embedding model produces similarity scores that are substantially worse than PPMI on standard benchmarks, that is a strong signal that something is wrong with your training setup, not an inherent limitation of the distributional approach.

The influence of PMI extends into the transformer era. Attention mechanisms in transformers can be understood as a learned, soft version of context weighting, where the attention weights play a role analogous to PMI: they determine which context positions are relevant to each target position. The intuition that "which words appear together more than chance would predict" is meaningful has carried through from 1990s corpus linguistics into the core architectural decisions of modern language models.

Pointwise Mutual Information converts raw co-occurrence counts into normalized association scores by asking how much more often two words appear together than chance would predict. The core formula compares the observed joint probability against the product of the marginal probabilities, using a logarithm to produce an additive scale where zero means independence.

The key ideas to remember:

• PMI = 0 means no association beyond chance. PMI > 0 means positive beyond-chance association. PMI < 0 means co-occurrence less than chance, often due to sparsity.
• PPMI replaces negative values with zero, avoiding the $-\infty$ problem from unseen pairs and producing vectors suitable for downstream similarity tasks.
• Shifted PPMI subtracts $\log_2 k$ before clipping, reducing the influence of weak co-occurrences and connecting to the implicit objective of word2vec skip-gram training.
• Rare-word bias is PMI's main practical weakness: low-frequency words with any co-occurrence can score very high, so minimum count thresholds or smoothing are important in practice.
• Collocation extraction is a natural PMI application, since high PMI scores identify word pairs that form meaningful multi-word expressions by appearing far more often than chance.
• Broader applications include word-document weighting, feature selection for NER, and semantic orientation scoring for sentiment analysis.
• The distributional hypothesis underlies all PMI-based methods: words used in similar contexts receive similar PPMI vectors, enabling semantic similarity to be computed from corpus statistics alone.

In the next chapter, we apply Singular Value Decomposition to the PPMI matrix, compressing its high-dimensional sparse structure into dense, low-dimensional vectors that often capture semantic relationships even more cleanly and generalize better to word pairs not seen in training.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about Pointwise Mutual Information.