Pointwise Mutual Information › Summary

Pointwise Mutual Information converts raw co-occurrence counts into normalized association scores by asking how much more often two words appear together than chance would predict. The core formula compares the observed joint probability against the product of the marginal probabilities, using a logarithm to produce an additive scale where zero means independence.

The key ideas to remember:

• PMI = 0 means no association beyond chance. PMI > 0 means positive beyond-chance association. PMI < 0 means co-occurrence less than chance, often due to sparsity.
• PPMI replaces negative values with zero, avoiding the $-\infty$ problem from unseen pairs and producing vectors suitable for downstream similarity tasks.
• Shifted PPMI subtracts $\log_2 k$ before clipping, reducing the influence of weak co-occurrences and connecting to the implicit objective of word2vec skip-gram training.
• Rare-word bias is PMI's main practical weakness: low-frequency words with any co-occurrence can score very high, so minimum count thresholds or smoothing are important in practice.
• Collocation extraction is a natural PMI application, since high PMI scores identify word pairs that form meaningful multi-word expressions by appearing far more often than chance.
• Broader applications include word-document weighting, feature selection for NER, and semantic orientation scoring for sentiment analysis.
• The distributional hypothesis underlies all PMI-based methods: words used in similar contexts receive similar PPMI vectors, enabling semantic similarity to be computed from corpus statistics alone.

In the next chapter, we apply Singular Value Decomposition to the PPMI matrix, compressing its high-dimensional sparse structure into dense, low-dimensional vectors that often capture semantic relationships even more cleanly and generalize better to word pairs not seen in training.