Pointwise Mutual Information › Smoothing and Practical Considerations

The basic PPMI formula works well for large, balanced corpora, but several practical issues arise in real deployments that require additional engineering decisions.

The simplest and most effective way to control the rare-word problem is to apply a minimum frequency threshold before computing PPMI. Any word that appears fewer than $k$ times in the corpus is excluded from the vocabulary, and any co-occurrence pair with fewer than $m$ joint occurrences is treated as zero. Typical values are $k = 5$ for vocabulary thresholds and $m = 2$ or $m = 5$ for co-occurrence thresholds.

This approach has several benefits. It reduces the vocabulary size, which reduces memory requirements for the co-occurrence matrix. It eliminates the highest-variance PMI estimates, those based on only one or two co-occurrence events. And it focuses the representation on the part of the vocabulary with sufficient evidence for reliable estimates.

The main cost is coverage: words that appear fewer than five times in the corpus are excluded entirely, which can be significant for technical domains with specialized vocabulary or for morphologically rich languages where every inflected form is counted separately.

Laplace smoothing (also called add-$\alpha$ smoothing) addresses the zero-count problem in a more principled way than simply discarding rare pairs. Instead of working with the raw counts $\text{count}(w, c)$, you add a small pseudocount $\alpha$ to every cell:

$$\text{count}_\alpha(w, c) = \text{count}(w, c) + \alpha$$

With $\alpha = 1$ (standard Laplace smoothing) or $\alpha < 1$ (Lidstone smoothing), every word pair gets a small baseline probability even if it was never observed. This prevents $-\infty$ from appearing in the PMI computation and reduces the variance of estimates for rare pairs.

The practical effect is that the PPMI matrix becomes denser: pairs that would have been zero now have small positive values. This can improve performance on downstream tasks that use the PPMI matrix as input to a learning algorithm, because the algorithm receives a more complete picture of the distributional structure.

Some PPMI implementations apply sublinear scaling to the co-occurrence counts before computing probabilities. One common variant counts each co-occurrence event with weight $1 / d$ where $d$ is the distance between the two words within the window. Words that are immediately adjacent receive full weight, while words at the far edge of the window receive reduced weight.

This position-weighted counting better reflects the intuition that nearby words are more informative context than distant ones. The resulting PPMI values place more weight on tight, adjacent associations and less on loose, distant ones. GloVe uses a related but distinct weighting function: the weight decreases with distance but is bounded by a maximum value of 1 to avoid over-weighting very frequent immediate neighbors.

Levy and Goldberg also noted that word2vec implicitly raises context word frequencies to the power $3/4$ before using them as negative sampling probabilities. This smoothing reduces the dominance of very high-frequency context words in the model's training signal. The equivalent for PPMI is to raise the context word counts to the power $\alpha$ (typically $\alpha = 0.75$) before computing column marginals:

$$P_\alpha(c) = \frac{\text{count}(c)^\alpha}{\sum_{c'} \text{count}(c')^\alpha}$$

where:

• $\alpha$: the smoothing exponent, typically 0.75
• $\text{count}(c)^\alpha$: the smoothed count for context word $c$

The modified PPMI using this smoothed marginal is:

$$\text{PPMI}_\alpha(w, c) = \max\Bigl(0,\; \log_2 \frac{P(w, c)}{P(w) \cdot P_\alpha(c)}\Bigr)$$

This smoothing reduces the raw PMI of pairs involving very high-frequency context words, bringing the PPMI formulation closer to the implicit objective of word2vec with negative sampling. In practice, this variant consistently outperforms standard PPMI on word similarity benchmarks.