Pointwise Mutual Information › PMI for Broader Distributional Semantics

PMI is not limited to word-word co-occurrence. The same formula applies whenever you want to measure the association between any two categorical variables from count data, and this generality has made it useful across a wide range of NLP applications.

In information retrieval, PMI is used to weight word-document associations. Instead of TF-IDF (which combines term frequency with inverse document frequency), you can compute PMI between each word and each document:

$$\text{PMI}(w, d) = \log_2 \frac{P(w, d)}{P(w) \cdot P(d)}$$

where $P(w, d)$ is the probability of observing word $w$ in document $d$, $P(w)$ is the overall word frequency, and $P(d)$ is the document's relative length. High PMI for (word, document) pairs identifies words that are distinctive of specific documents, similar in spirit to TF-IDF but derived from a different theoretical foundation.

Named entity recognition and relation extraction systems have historically used PMI to identify features that are distinctive of entity classes. For example, you might compute PMI between each word and the label "PERSON" based on annotated training data. Words with high PMI for "PERSON" (like "Mr.," "Ms.," capitalized names) become strong features for the classifier, while words with PMI near zero are not discriminative for that class.

Turney (2002) used PMI to compute the semantic orientation of adjectives for sentiment analysis, in a method that predates the deep learning era but remains influential. The idea is to compute the PMI between each adjective and two reference words representing positive sentiment ("excellent") and negative sentiment ("poor"), then take the difference:

$$\text{SO}(w) = \text{PMI}(w, \text{excellent}) - \text{PMI}(w, \text{poor})$$

where:

• $\text{SO}(w)$: the semantic orientation score for word $w$
• $\text{PMI}(w, \text{excellent})$: how strongly $w$ associates with positive sentiment
• $\text{PMI}(w, \text{poor})$: how strongly $w$ associates with negative sentiment

An adjective with positive SO ("wonderful," "brilliant") co-occurs more strongly with "excellent" than with "poor." An adjective with negative SO ("terrible," "mediocre") has the reverse pattern. This simple approach achieved surprisingly strong performance on review classification tasks. This shows that PMI can capture meaningful semantic properties without any supervised training.