InteractivePointwise Mutual Information: Measuring Word Associations in NLP

Pointwise Mutual InformationLink Copied

Raw co-occurrence counts tell us how often words appear together, but they have a fundamental problem: frequent words dominate everything. The word "the" co-occurs with nearly every other word, not because it has a special relationship with them, but simply because it appears everywhere. How do we separate meaningful associations from mere frequency effects?

Pointwise Mutual Information (PMI) solves this problem by asking a different question: does this word pair co-occur more than we'd expect by chance? Instead of counting raw co-occurrences, PMI measures the surprise of seeing two words together. When "New" and "York" appear together far more often than their individual frequencies would predict, PMI captures that strong association. When "the" and "dog" appear together only as often as expected, PMI correctly identifies this as an unremarkable pairing.

This chapter derives the PMI formula from probability theory, shows how it transforms co-occurrence matrices into more meaningful representations, and demonstrates its practical applications from collocation extraction to improved word vectors.

The Problem with Raw CountsLink Copied

Let's start by understanding why raw co-occurrence counts are problematic. Consider a corpus about animals and food:

The word "the" appears 18 times, dwarfing content words like "cat" (5 times) or "new" (3 times). In a raw co-occurrence matrix, "the" will have high counts with almost everything, obscuring the meaningful associations we care about.

The raw counts show "the" as the top co-occurrence for "cat," but this tells us nothing about cats. The word "the" appears near everything. We need a measure that accounts for how often we'd expect words to co-occur given their individual frequencies.

The PMI FormulaLink Copied

The problem we identified, that frequent words dominate co-occurrence counts, stems from a fundamental issue: raw counts conflate two distinct phenomena. When "the" appears near "cat" 10 times, is that because "the" and "cat" have a special relationship, or simply because "the" appears near everything 10 times? To separate genuine associations from frequency effects, we need to ask a different question: how does the observed co-occurrence compare to what we'd expect if the words were unrelated?

Pointwise Mutual Information answers this question directly. Instead of asking "how often do these words appear together?" PMI asks "do these words appear together more or less than chance would predict?"

The Intuition: Observed vs ExpectedLink Copied

Consider two words, $w$ and $c$. If they have no special relationship, their co-occurrence should follow a simple pattern: the probability of seeing them together should equal the product of their individual probabilities. This is the definition of statistical independence.

For example, if "cat" appears in 5% of contexts and "mouse" appears in 2% of contexts, and they're independent, we'd expect "cat" and "mouse" to co-occur in roughly $0.05 \times 0.02 = 0.1\%$ of word pairs. If they actually co-occur in 1% of pairs, that's 10 times more than expected. This excess reveals a genuine association.

PMI formalizes this comparison by taking the ratio of observed to expected co-occurrence:

Let's unpack each component of this formula:

• Numerator $P(w, c)$: The joint probability, measuring how often words $w$ and $c$ actually co-occur in the corpus.

• Denominator $P(w) \cdot P(c)$: The expected probability under independence. If the words were unrelated, this is how often they would co-occur by chance.

• The ratio: When observed equals expected, the ratio is 1. When observed exceeds expected, the ratio is greater than 1. When observed falls short of expected, the ratio is less than 1.

• The logarithm: Taking $\log_2$ converts multiplicative relationships to additive ones. A ratio of 2 (twice as often as expected) becomes PMI of 1. A ratio of 4 becomes PMI of 2. This logarithmic scale makes PMI values easier to interpret and compare.

Why the Logarithm?Link Copied

The logarithm serves three purposes. First, it converts ratios to differences: PMI of 2 means "4 times more than expected," while PMI of 1 means "2 times more." The additive scale is more intuitive.

Second, it creates symmetry around zero. Positive PMI indicates attraction (words co-occur more than expected), negative PMI indicates repulsion (words co-occur less than expected), and zero indicates independence.

Third, it connects to information theory. PMI measures how much information observing one word provides about the other. When two words are strongly associated, seeing one tells you a lot about whether you'll see the other.

From Probabilities to CountsLink Copied

In practice, we don't know the true probabilities. We estimate them from corpus counts. Let $\#(w, c)$ denote how many times words $w$ and $c$ co-occur, and let $N$ be the total number of word-context pairs in our co-occurrence matrix.

The probability estimates follow naturally:

This is the fraction of all word pairs that are specifically the $(w, c)$ pair.

This is the fraction of all pairs where the target word is $w$. The notation $\#(w, *)$ means "sum over all context words," which is simply the row sum for word $w$ in the co-occurrence matrix.

Similarly, this is the column sum for context word $c$, representing how often $c$ appears as a context for any word.

The Practical FormulaLink Copied

Substituting these estimates into the PMI formula:

Notice that $N$ appears in both numerator and denominator. With some algebra, we can simplify:

This is the formula we implement: multiply the co-occurrence count by the total, divide by the product of marginals, and take the logarithm. Each term has a clear interpretation:

• $\#(w, c)$: How often $w$ and $c$ actually co-occur
• $N$: Total co-occurrences in the matrix
• $\#(w, *)$: How often $w$ appears with any context (row sum)
• $\#(*, c)$: How often $c$ appears as context for any word (column sum)

Implementing PMILink Copied

Let's translate this formula into code. The implementation follows the mathematical derivation closely, computing each component step by step.

The key insight in this implementation is computing the expected counts efficiently. Rather than looping through all word pairs, we use matrix multiplication: row_sums @ col_sums produces an outer product where each cell $(i, j)$ contains $\#(w_i, *) \cdot \#(*, c_j)$. Dividing by $N$ gives us the expected count for each pair.

This scatter plot visualizes the fundamental idea behind PMI: comparing what we observe to what we'd expect. Word pairs above the diagonal have positive PMI (they co-occur more than chance predicts), while pairs below have negative PMI.

The transformation is dramatic. Raw counts ranked "the" as the top co-occurrence for "cat," but PMI reveals a different picture. Words that specifically associate with "cat," rather than appearing everywhere, now rise to the top. The word "the" has low or negative PMI because its high frequency means it co-occurs with "cat" about as often as we'd expect by chance.

Interpreting PMI as Association StrengthLink Copied

The logarithmic scale of PMI creates a natural interpretation centered on zero. Because we're measuring $\log_2(\text{observed} / \text{expected})$, the value tells us directly how the actual co-occurrence compares to the baseline of independence:

• PMI > 0: The words co-occur more than expected. A PMI of 1 means twice as often as chance; PMI of 2 means four times as often; PMI of 3 means eight times. The higher the value, the stronger the positive association.

• PMI = 0: The words co-occur exactly as expected under independence. There's no special relationship, either attractive or repulsive.

• PMI < 0: The words co-occur less than expected. They tend to avoid each other. A PMI of -1 means half as often as chance would predict.

A Worked Example: Computing PMI Step by StepLink Copied

To solidify understanding, let's walk through a complete PMI calculation by hand. We'll compute the PMI between "cat" and "mouse," two words we'd expect to have a strong association.

The calculation proceeds in three stages:

1. Gather the raw counts from our co-occurrence matrix
2. Compute the expected count under the assumption of independence
3. Calculate PMI as the log ratio of observed to expected

The positive PMI confirms our linguistic intuition: "cat" and "mouse" have a genuine association in language. They appear together in contexts about predator-prey relationships, children's stories, and idiomatic expressions far more often than their individual frequencies would suggest.

This is exactly the kind of meaningful relationship PMI is designed to uncover. Raw counts would have told us that "the" co-occurs with "cat" more often than "mouse" does, simply because "the" is everywhere. PMI cuts through this noise by asking the right question: not "how often?" but "how much more than expected?"

The Problem with Negative PMILink Copied

While PMI can be negative (indicating words that avoid each other), negative values cause practical problems. Most word pairs never co-occur at all, giving them PMI of negative infinity. Even pairs that co-occur rarely get large negative values.

The histogram reveals a key characteristic of PMI: many word pairs have negative values, meaning they co-occur less than expected. This asymmetry, combined with the issues below, motivates the PPMI transformation.

Negative PMI values are problematic for several reasons:

1. Unreliable estimates: Low co-occurrence counts produce noisy PMI values. A word pair that co-occurs once when we expected two has PMI of -1, but this could easily be sampling noise.

2. Asymmetric information: Knowing that words don't co-occur is less informative than knowing they do. The absence of co-occurrence could mean many things.

3. Computational issues: Large negative values dominate distance calculations and can destabilize downstream algorithms.

Positive PMI (PPMI)Link Copied

The standard solution is Positive PMI (PPMI), which simply clips negative values to zero:

The PPMI matrix is much sparser than the raw co-occurrence matrix. Only word pairs with genuine positive associations retain non-zero values. This sparsity is a feature, not a bug: it means we've filtered out the noise of random co-occurrences.

Shifted PPMI VariantsLink Copied

While PPMI works well, researchers have developed variants that address specific issues. The most important is Shifted PPMI, which subtracts a constant before clipping:

where $k$ is a shift parameter, typically between 1 and 15.

Why shift? The shift acts as a threshold for what counts as a "meaningful" association. With $k=1$ (no shift), any positive PMI is retained. With $k=5$, only word pairs that co-occur at least 5 times more than expected survive.

The visualization shows the trade-off clearly: higher shift values produce sparser matrices by filtering out weaker associations. A common choice is $k=5$, which corresponds to Word2Vec's default of 5 negative samples.

The connection to Word2Vec is notable: Levy and Goldberg (2014) showed that Word2Vec's skip-gram model with negative sampling implicitly factorizes a shifted PMI matrix with $k$ equal to the number of negative samples. This theoretical connection explains why PMI-based methods and neural embeddings often produce similar results.

PMI vs Raw Counts: A ComparisonLink Copied

Let's directly compare how raw counts and PPMI rank word associations. We'll use a larger corpus to see clearer patterns.

The PPMI rankings are more semantically meaningful. Raw counts are dominated by frequent function words, while PPMI surfaces content words with genuine topical associations.

PMI Matrix PropertiesLink Copied

PPMI matrices have several useful properties that make them well-suited for downstream NLP tasks.

SparsityLink Copied

PPMI matrices are highly sparse because most word pairs don't have positive associations. This sparsity enables efficient storage and computation.

The sparsity visualization makes the filtering effect of PPMI immediately apparent. The raw matrix has entries wherever words co-occur at all, while the PPMI matrix retains only the genuinely positive associations. This much smaller subset captures the meaningful relationships.

SymmetryLink Copied

For symmetric context windows (looking the same distance left and right), the co-occurrence matrix is symmetric, and so is the PPMI matrix: $\text{PPMI}(w, c) = \text{PPMI}(c, w)$.

This symmetry means the association between words is bidirectional: if "neural" has high PPMI with "networks," then "networks" has equally high PPMI with "neural."

Row Vectors as Word RepresentationsLink Copied

Each row of the PPMI matrix can serve as a word vector. Words with similar PPMI profiles (similar rows) tend to have similar meanings.

The similarity scores capture semantic relationships learned purely from co-occurrence patterns. Words appearing in similar contexts cluster together in the PPMI vector space, enabling applications like finding related terms or detecting semantic categories.

The 2D projection reveals the semantic structure captured by PPMI vectors. Even with our small corpus, related words cluster together: machine learning terms form one group, data-related terms another. This is the distributional hypothesis in action. Words with similar meanings appear in similar contexts, leading to similar PPMI vectors.

Collocation Extraction with PMILink Copied

One of PMI's most practical applications is identifying collocations: word combinations that occur together more than chance would predict. Collocations include compound nouns ("ice cream"), phrasal verbs ("give up"), and idiomatic expressions ("kick the bucket").

The top collocations are meaningful multi-word expressions like "machine learning," "neural networks," and "language models." PMI successfully identifies these as units that co-occur far more than chance would predict. The high PMI scores (often above 4) indicate these word pairs appear together 16 or more times more frequently than their individual frequencies would suggest.

Implementation: Building a Complete PPMI PipelineLink Copied

Let's put everything together into a complete, reusable implementation for computing PPMI matrices from text.

The vectorizer successfully builds PPMI word vectors and finds semantically related words. The similarity scores reflect how often words share the same contexts in the corpus. This encapsulated implementation can be reused across different corpora and easily integrated into larger NLP pipelines.

Limitations and When to Use PMILink Copied

While PMI is powerful, it has limitations you should understand.

Sensitivity to Low CountsLink Copied

PMI estimates are unreliable for rare word pairs. A word pair that co-occurs once when we expected 0.5 co-occurrences gets PMI of 1, but this could easily be noise. The standard solution is to require minimum co-occurrence counts before computing PMI.

Requiring a minimum co-occurrence count filters out potentially spurious associations that could arise from sampling noise, producing more reliable PMI estimates at the cost of discarding some valid but rare associations.

Bias Toward Rare WordsLink Copied

PMI tends to give high scores to rare word pairs. If a rare word appears only in specific contexts, it gets high PMI with those contexts even if the association is coincidental. Shifted PPMI helps by raising the threshold for positive associations.

The downward trend confirms the rare word bias: words with fewer total co-occurrences tend to achieve higher maximum PMI scores. This happens because rare words have limited contexts, making each co-occurrence count proportionally more.

Computational CostLink Copied

For large vocabularies, PMI matrices become enormous. A 100,000-word vocabulary produces a 10-billion-cell matrix. While the matrix is sparse after PPMI transformation, the intermediate computations can be expensive. Practical implementations use sparse matrix formats and streaming algorithms.

When to Use PMILink Copied

PMI and PPMI are excellent choices when:

• You need interpretable association scores between words
• You're extracting collocations or multi-word expressions
• You want sparse, high-dimensional word vectors as input to other algorithms
• You need a baseline to compare against neural embeddings
• Computational resources are limited (no GPU required)

Neural methods like Word2Vec often outperform PPMI for downstream tasks, but the difference is smaller than you might expect. For many applications, PPMI provides a strong, interpretable baseline.

SummaryLink Copied

Pointwise Mutual Information transforms raw co-occurrence counts into meaningful association scores by comparing observed co-occurrence to what we'd expect under independence.

Key concepts:

• PMI formula: $\text{PMI}(w, c) = \log_2 \frac{P(w, c)}{P(w) \cdot P(c)}$ measures how much more (or less) two words co-occur than chance would predict

• Positive PMI (PPMI): Clips negative values to zero, keeping only positive associations. This produces sparse matrices that work well with machine learning algorithms.

• Shifted PPMI: Subtracts $\log_2 k$ before clipping, filtering out weak associations. Connected theoretically to Word2Vec's negative sampling.

• PMI interpretation: Positive PMI means words co-occur more than expected (strong association). Zero means independence. Negative means avoidance.

Practical applications:

• Collocation extraction: Finding meaningful multi-word expressions
• Word similarity: Using PPMI vectors with cosine similarity
• Feature weighting: PPMI as a preprocessing step before dimensionality reduction

Key parameters:

The next chapter shows how to reduce the dimensionality of PPMI matrices using Singular Value Decomposition, producing dense vectors that capture the essential structure in fewer dimensions.

QuizLink Copied

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about Pointwise Mutual Information and word associations.