Pointwise Mutual Information › Worked Example: Computing PMI by Hand

Let's work through a small example to build intuition. Consider a tiny corpus of four sentences:

"The cat chases mice." "The dog chases cats." "Cats and dogs are common pets." "Mice fear cats."

Using a window size of 1 (immediate neighbors only), we count co-occurrences and compute PMI for the pair (cat, chases).

Suppose after counting we have:

• $\text{count}(\text{cat}, \text{chases}) = 2$
• $\text{count}(\text{cat}) = 4$ (total number of co-occurrence events in which "cat" participates)
• $\text{count}(\text{chases}) = 3$ (total number of co-occurrence events in which "chases" participates)
• $N = 20$ (total co-occurrence events across all word pairs)

The estimated probabilities are:

$$\begin{aligned} P(\text{cat}, \text{chases}) &= \frac{2}{20} = 0.10 \\[4pt] P(\text{cat}) &= \frac{4}{20} = 0.20 \\[4pt] P(\text{chases}) &= \frac{3}{20} = 0.15 \end{aligned}$$

The expected joint probability under the independence assumption would be:

$$P(\text{cat}) \cdot P(\text{chases}) = 0.20 \times 0.15 = 0.030$$

Computing PMI:

$$\text{PMI}(\text{cat}, \text{chases}) = \log_2 \frac{0.10}{0.030} = \log_2 3.33 \approx 1.74$$

A PMI of 1.74 bits means "cat" and "chases" co-occur about 3.3 times more often than chance would predict. This represents a meaningful association: the corpus provides evidence that cats chase things.

Now compare this with the pair (the, chases). Suppose "the" appears so frequently that $P(\text{the}) = 0.25$, but co-occurs with "chases" only about as often as expected by chance. Under near-independence, the ratio is close to 1 and:

$$\text{PMI}(\text{the}, \text{chases}) = \log_2 \frac{P(\text{the}, \text{chases})}{P(\text{the}) \cdot P(\text{chases})} \approx \log_2 1.0 = 0$$

PMI correctly reveals that "the" and "chases" have no special relationship, even though "the" may appear many times near "chases" in absolute terms.

Let's also verify the count-based formula directly to make sure the algebra is clear. From the derivation above:

$$\text{PMI}(w, c) = \log_2 \frac{\text{count}(w, c) \cdot N}{\text{count}(w) \cdot \text{count}(c)}$$

Plugging in the values from the (cat, chases) example:

$$\begin{aligned} \text{PMI}(\text{cat}, \text{chases}) &= \log_2 \frac{2 \times 20}{4 \times 3} \\[4pt] &= \log_2 \frac{40}{12} \\[4pt] &= \log_2 3.33 \\[4pt] &\approx 1.74 \end{aligned}$$

The count-based formula gives the same result as the probability-based formula, as expected. In code, you will typically use the count-based form because it avoids explicitly computing probabilities and works directly with the co-occurrence matrix.

Now suppose we also compute PMI for a pair with only one co-occurrence, where both words are relatively rare. Say $\text{count}(w_1, w_2) = 1$, $\text{count}(w_1) = 2$, $\text{count}(w_2) = 2$, and $N = 20$:

$$\begin{aligned} \text{PMI}(w_1, w_2) &= \log_2 \frac{1 \times 20}{2 \times 2} \\[4pt] &= \log_2 5 \\[4pt] &\approx 2.32 \end{aligned}$$

This PMI of 2.32 is higher than the (cat, chases) PMI of 1.74, even though the evidence is much thinner: a single co-occurrence vs. two. This illustrates the rare-word bias. PPMI retains this value as-is, which is why minimum count thresholds are important in practice.

And for a pair that never co-occurs, $\text{count}(w, c) = 0$:

$$\text{PMI}(w, c) = \log_2 \frac{0 \cdot N}{\text{count}(w) \cdot \text{count}(c)} = \log_2 0 = -\infty$$

PPMI converts this to 0, handling the undefined case cleanly.