Pointwise Mutual Information › Limitations and Impact

PMI has been enormously influential in NLP, but it comes with several well-understood limitations that practitioners need to keep in mind.

The most significant practical limitation is sensitivity to rare events. When a word pair appears only once or twice, the PMI estimate is unreliable. A pair might score very high simply because one or both words are rare in general, making any co-occurrence appear remarkable by chance. This is why PPMI benefits from additional smoothing or minimum count thresholds in production systems. The problem worsens with larger vocabularies: a vocabulary of 100,000 words produces a $100{,}000 \times 100{,}000$ matrix with 10 billion cells, the vast majority of which will be empty, and the non-zero cells will often have very small counts.

PMI also ignores word order within the context window. The pair (cat, chased) scores the same regardless of whether the cat is doing the chasing or being chased. For many tasks this is fine, but for understanding more detailed semantic relationships like agent-patient roles, you would need directional weighting or syntactic context. Similarly, all positions within the window contribute equally by default. A word appearing immediately adjacent to the target is treated the same as a word appearing at the window boundary, even though immediate neighbors are usually more informative. Position-weighted variants address this but add complexity.

A third limitation is the binary nature of co-occurrence windows. PMI measures whether two words appeared within a window, but it does not distinguish between a pair that appears ten times at distance 1 and a pair that appears ten times at distances 1 through 5. More sophisticated approaches, including GloVe and neural word embeddings, use weighted or richer context representations that capture more positional information.

Despite these limitations, PMI made several lasting contributions to NLP. It provided a principled, interpretable way to measure word association, replacing simple frequency thresholds and ad hoc weighting schemes. PPMI matrices served as the foundation for Latent Semantic Analysis (LSA) and its successors, proving that dense, low-dimensional word representations could capture meaningful semantic properties. When Levy and Goldberg showed that word2vec implicitly factorizes a shifted PMI matrix, it revealed that neural word embeddings and classical distributional models are mathematically related, unifying decades of research under a common framework.

For collocation extraction specifically, PMI remains competitive with more sophisticated methods. The association between a trigram like "New York City" or a bigram like "kick the bucket" is well-captured by PMI, which correctly identifies that these combinations appear far more often than chance would predict. This makes PMI a practical tool in applications ranging from phrase detection in search engines to terminology extraction in specialized domains.

PMI also remains widely used as a baseline and evaluation tool. When developing new word embedding techniques, comparing against PPMI vectors provides an interpretable reference point. Since PPMI vectors are sparse and directly interpretable (each dimension corresponds to a specific context word), they make it easy to audit what a model has learned and diagnose failure modes. If your neural embedding model produces similarity scores that are substantially worse than PPMI on standard benchmarks, that is a strong signal that something is wrong with your training setup, not an inherent limitation of the distributional approach.

The influence of PMI extends into the transformer era. Attention mechanisms in transformers can be understood as a learned, soft version of context weighting, where the attention weights play a role analogous to PMI: they determine which context positions are relevant to each target position. The intuition that "which words appear together more than chance would predict" is meaningful has carried through from 1990s corpus linguistics into the core architectural decisions of modern language models.