Pointwise Mutual Information › Deriving the PMI Formula

PMI comes from information theory, specifically from the concept of mutual information between random variables. Mutual information asks how much knowing one variable tells you about another. It quantifies the reduction in uncertainty about one variable when you know the value of another. Pointwise mutual information applies this question to a specific pair of outcomes rather than to entire distributions.

Think of each word occurrence as a random event. When you sample a position in the corpus at random, you draw a word. Define:

• $P(w)$: the probability of randomly sampling word $w$ from the corpus
• $P(c)$: the probability of randomly sampling context word $c$
• $P(w, c)$: the probability of observing the word pair $(w, c)$ together in a co-occurrence event

If $w$ and $c$ were completely independent, we would expect their joint probability to equal the product of their marginal probabilities:

$$P(w, c)_{\text{independent}} = P(w) \cdot P(c)$$

This is simply the definition of statistical independence. Two events are independent if and only if their joint probability factors into the product of the marginals. If you know that "bank" and "deposit" are independent, then knowing "bank" appeared tells you nothing about whether "deposit" appeared nearby.

PMI measures the log-ratio of the observed joint probability to this independent baseline:

$$\text{PMI}(w, c) = \log_2 \frac{P(w, c)}{P(w) \cdot P(c)}$$

where:

• $P(w, c)$: the observed probability of word $w$ appearing with context $c$
• $P(w)$: the marginal probability of word $w$ across all contexts
• $P(c)$: the marginal probability of context $c$ across all co-occurrences
• $\log_2$: logarithm base 2, so PMI values are expressed in bits

When the observed probability equals the expected probability under independence, the ratio is 1 and $\log_2(1) = 0$. When the words co-occur more than chance predicts, the ratio exceeds 1 and PMI is positive. When words co-occur less than chance predicts, the ratio falls below 1 and PMI is negative.

The logarithm serves several important roles. First, it converts the ratio from a multiplicative scale to an additive one, which is easier to work with mathematically and algorithmically. Second, it gives PMI a natural interpretation in information theory: a PMI of 1 bit means that observing $w$ doubles your probability estimate for $c$. Third, the logarithm makes PMI symmetric in the sense that $\text{PMI}(w, c) = \text{PMI}(c, w)$, which you can verify algebraically since the joint probability and the product of marginals are both symmetric.

In practice, we estimate these probabilities from a co-occurrence matrix. Given a corpus with $N$ total co-occurrence events:

• $P(w) = \dfrac{\text{count}(w)}{N}$, where $\text{count}(w)$ is the total number of co-occurrence events involving word $w$
• $P(c) = \dfrac{\text{count}(c)}{N}$, where $\text{count}(c)$ is the total number of co-occurrence events involving context $c$
• $P(w, c) = \dfrac{\text{count}(w, c)}{N}$, where $\text{count}(w, c)$ is the number of times $w$ and $c$ co-occur within the context window

Substituting these into the PMI formula and simplifying:

$$\begin{aligned} \text{PMI}(w, c) &= \log_2 \frac{P(w, c)}{P(w) \cdot P(c)} \\[6pt] &= \log_2 \frac{\dfrac{\text{count}(w, c)}{N}}{\dfrac{\text{count}(w)}{N} \cdot \dfrac{\text{count}(c)}{N}} \\[6pt] &= \log_2 \frac{\text{count}(w, c) \cdot N}{\text{count}(w) \cdot \text{count}(c)} \end{aligned}$$

This final form is what you implement in code: multiply the raw co-occurrence count by the total event count $N$, then divide by the product of the two marginal counts. The logarithm converts this ratio into an additive scale where zero means independence. Notice that $N$ cancels from the marginal probabilities but remains in the joint probability, which is why it appears in the numerator of the count-based formula. The ratio $\text{count}(w,c) \cdot N$ in the numerator can be understood as the observed count scaled up to match the units of the denominator $\text{count}(w) \cdot \text{count}(c)$.

PMI values have clear, intuitive meanings that map directly onto the concept of statistical association:

• PMI = 0: The words appear together exactly as often as chance predicts. No evidence of association beyond their individual frequencies.
• PMI > 0: The words appear together more than chance predicts. Higher values mean stronger, more specific association. A PMI of 2 bits means the pair co-occurs four times more often than chance.
• PMI < 0: The words appear together less than chance predicts. They tend to avoid each other, though this is often an artifact of data sparsity rather than actual repulsion.
• PMI approaching $+\infty$: Occurs when two words almost always appear together but one or both are very rare. This can happen with highly specialized terminology or named entities in small corpora.
• PMI $= -\infty$: Occurs when $\text{count}(w, c) = 0$. The words never co-occur, making the log of zero undefined.

The $-\infty$ case is a serious practical problem. Many word pairs never co-occur, especially in large vocabularies with limited data. A vocabulary of 50,000 words yields a matrix with 2.5 billion cells, but even a large corpus might only fill a small fraction of them. This leads directly to the concept of Positive PMI.

A caveat is that PMI values for rare words can be very high even when based on flimsy evidence. If two words each appear only twice in the corpus but always together, their PMI is very high. That sounds like strong evidence, but two observations are not enough to establish an association. This statistical fragility for low-frequency pairs is one of PMI's main practical weaknesses and motivates the smoothing and thresholding strategies discussed later.

PMI is grounded in mutual information rather than being an ad hoc normalization. It has a direct connection to mutual information, one of the foundational quantities in information theory. Mutual information between two random variables $W$ and $C$ is defined as:

$$I(W; C) = \sum_{w} \sum_{c} P(w, c) \cdot \log_2 \frac{P(w, c)}{P(w) \cdot P(c)}$$

This is exactly a weighted average of PMI values, where the weights are the joint probabilities $P(w, c)$. Mutual information measures the total statistical dependency between $W$ and $C$ across all possible word and context pairs. Pointwise mutual information zooms in on a single pair $(w, c)$ and reports the contribution of that specific combination to the overall mutual information.

This connection matters for two reasons. First, it means PMI has a well-founded theoretical interpretation, rather than a purely heuristic one. Second, it means that maximizing PMI for specific pairs is related to, though not identical to, maximizing mutual information across all pairs, which is the objective of many representation learning algorithms.