Pointwise Mutual Information › Positive PMI (PPMI)

The most widely used variant of PMI is Positive PMI (PPMI), which replaces all negative values (including $-\infty$) with zero:

$$\text{PPMI}(w, c) = \max(0, \text{PMI}(w, c))$$

where:

• $\text{PMI}(w, c)$: the pointwise mutual information as computed above
• $\max(0, \cdot)$: clips the value at zero, replacing any negative or undefined result with 0

This single modification handles the zero-count problem cleanly: any word pair that never co-occurs has $\text{PMI} = -\infty$, and PPMI converts that to 0. PPMI also discards actual negative associations, treating "never seen together" and "seen together less often than chance" both as zero. This is a deliberate simplification: for most NLP tasks, we care about positive associations. A high PPMI score reliably indicates a real linguistic relationship, while negative scores are harder to interpret reliably, particularly because many negative values arise from data sparsity rather than systematic avoidance.

PPMI also has a valuable scale-invariance property. Doubling the corpus size does not change PPMI scores, because both the joint and marginal probabilities scale proportionally. Raw co-occurrence counts, by contrast, simply grow with corpus size, making cross-corpus comparisons unreliable.

There is also a practical engineering benefit to PPMI: it produces non-negative vectors. Many downstream applications, including certain dimensionality reduction algorithms and cosine similarity computations, benefit from or require non-negative inputs. A PPMI vector is a legitimate sparse non-negative vector where each entry represents the strength of a specific word-context association.

The decision to replace negative PMI values with zero deserves more explanation, because it might seem like we are discarding information. The key insight is that negative PMI values are epistemically much weaker than positive ones.

A positive PMI value says: "I observed this pair appearing together more often than chance would predict." That is a positive empirical signal. The higher the count, the more confident you can be.

A negative PMI value, in the range available from finite corpora, says only: "I did not observe this pair appearing together as often as chance would predict." But absence of evidence is weak evidence of absence, especially when the corpus is finite. If "bank" and "octopus" have never appeared in the same window in your corpus, that might be because they systematically avoid each other, or it might be because your corpus simply doesn't contain enough diverse text to cover every conceivable word combination. In a sufficiently large corpus, almost every word appears near almost every other word at least once, so the zero-count entries are an artifact of corpus size, not a meaningful linguistic fact.

This asymmetry in reliability is why PPMI is standard practice. Positive values are trustworthy signals; negative and zero values are uninformative.

PMI has a well-known bias toward rare words. If two words appear together only once but each word is itself rare in the corpus, their PMI can be extremely high even though the evidence from a single observation is weak and statistically unreliable. One co-occurrence of a rare pair is not enough to establish an association.

Conversely, very frequent words like "the" and "of" tend to have large absolute co-occurrence counts but low PMI, because their high individual frequencies make high co-occurrence expected. PMI correctly identifies that "the" has no special relationship with "bank" despite often appearing near it.

The result is that PMI rewards rare, specific word pairs and penalizes common words. This is usually what we want: we care more that "asylum" and "seeker" associate strongly than that "the" and "is" associate. But the rare-word bias can cause problems when the vocabulary contains many low-frequency terms with minimal evidence. A word that appears three times in the corpus, all three times near the same context word, will receive a very high PMI for that context, but the signal is statistically fragile.

In practice, imposing a minimum co-occurrence count threshold (discarding pairs with fewer than five or ten co-occurrences) substantially reduces this noise. More principled approaches include Laplace smoothing (adding a small pseudocount to all matrix cells before computing probabilities), discounting methods borrowed from language modeling, or using the shifted PPMI variant described next.

One principled fix for the rare-word bias is shifted PPMI (SPPMI), introduced by Levy and Goldberg (2014) in their influential analysis of word2vec. They showed that the skip-gram model with negative sampling implicitly factorizes a shifted PMI matrix. The shifted variant is:

$$\text{SPPMI}_k(w, c) = \max\bigl(0,\; \text{PMI}(w, c) - \log_2 k\bigr)$$

where:

• $k$: the number of negative samples, a hyperparameter matching word2vec's negative sampling count
• $\log_2 k$: the constant shift subtracted from each PMI value before clipping

With $k > 1$, the shift increases the threshold a word pair must exceed before contributing a nonzero value to the matrix. This effectively filters out weakly associated pairs and reduces the influence of rare co-occurrences that inflate standard PPMI. Common choices for $k$ are 1, 5, and 15, corresponding to typical negative sampling values used in word2vec training.

To understand why this shift makes sense, recall that word2vec with negative sampling trains a model to distinguish real context words from randomly sampled "noise" words. With $k$ noise samples per real context, the model learns to assign high scores to pairs that appear $k$ times more often than would be expected from random sampling. This is precisely the condition that $\text{PMI}(w,c) > \log_2 k$ captures. The shift aligns the PPMI threshold with the implicit threshold used by the neural model.

The connection Levy and Goldberg uncovered is theoretically significant: it means that training a neural skip-gram model is, in effect, doing matrix factorization on a shifted PMI matrix. The two approaches, one classical and statistical, one neural and gradient-based, are optimizing related objectives. This unification demonstrated that decades of distributional semantics research and modern neural word embeddings were more closely related than practitioners had assumed. The neural model's apparent advantage was not that it discovered fundamentally different structure in language, but that it used a different, often more effective, factorization method (stochastic gradient descent on the implicit matrix) combined with architectural choices like context smoothing and subsampling that are not part of the basic PPMI formulation.