Pointwise Mutual Information › Visualizing PMI

Visualizations make the effect of PMI normalization immediately clear in ways that tables cannot. The following plots examine the PPMI matrix structure, compare PMI against raw counts for a set of context words, and illustrate how different PMI shift values affect the resulting distribution.

A heatmap of the PPMI matrix for a focused subset of words reveals the association structure directly. By selecting words from different semantic domains, we can see whether PMI recovers the domain boundaries from raw text.

The heatmap clearly shows that words from the same semantic domain share high PPMI scores with each other. Animal-domain words ("cat," "dog," "mouse") show mutual associations, while financial words ("bank," "fees," "loan") form their own cluster. The word "bank" shows some crossover. This reflects its use in both financial and geographic contexts in the corpus.

Shifted PPMI with different values of $k$ changes the distribution of positive associations. Larger $k$ raises the threshold, retaining only the strongest associations and pruning pairs whose PMI exceeds independence only by a small margin.

Standard PPMI ($k=1$) retains all positive associations, including weakly associated pairs near zero. Shifted PPMI ($k=5$) discards these borderline pairs, retaining only word combinations with strong, reliable co-occurrence evidence. The trade-off is coverage versus precision: $k=1$ covers more pairs but includes more noise; larger $k$ is more conservative but more reliable.

In practice, choosing $k$ depends on the downstream task. For collocation extraction, where precision matters more than recall, a larger shift is appropriate. For constructing word vectors for a downstream similarity task, a smaller shift preserves more information about the distributional structure.

This visualization makes the ranking shift from raw counts to PPMI concrete for a set of target context words, confirming the analytical argument with an empirical comparison.