Co-occurrence Matrices › From Raw Counts to Meaningful Representations

Raw co-occurrence counts are only the starting point of distributional semantics. The matrix you build by counting co-occurrences is valuable, but it has well-known failure modes that make it unsuitable for direct use in most semantic tasks. Understanding these failure modes and how they are addressed is essential context for the chapters that follow.

The dominant failure mode of raw co-occurrence matrices is function word inflation. Words like "the," "of," "in," "and," and "a" appear near almost every content word in the language. Their co-occurrence counts with all words are high, not because they are semantically related to those words, but because they are grammatically pervasive. This means that if you measure the cosine similarity between any two words using raw co-occurrence counts, the similarity will be strongly inflated by the shared high counts with function words.

Consider asking: are "dog" and "cat" more similar to each other than "dog" is to "algorithm"? Intuitively yes, obviously. But in the raw co-occurrence matrix, both "cat" and "algorithm" will have high counts with "the," "in," "a," and other function words. These shared high counts can dominate the cosine similarity computation and obscure the semantic differences. The signal from meaningful co-occurrences (dog-cat sharing "pet," "animal," "chase") is buried under the noise from function word co-occurrences that both words share with every other word.

The solution is Pointwise Mutual Information (PMI), which rescales co-occurrence counts to measure statistical association rather than raw frequency. PMI asks: how much more often do words $i$ and $j$ co-occur than you would expect if they were independent? Words that co-occur only as often as expected by chance get a PMI of zero; words that co-occur more than expected get a positive PMI; words that co-occur less than expected get a negative PMI. We cover this in detail in the next chapter.

Even with PMI weighting, the word vectors produced by co-occurrence matrices are in a very high-dimensional space. Each word is represented as a vector of length $V$, where $V$ might be 50,000 or 100,000. Most of these dimensions are zero or very small. Computing cosine similarity in such high-dimensional spaces is computationally expensive, and the curse of dimensionality means that distances become increasingly concentrated as dimensionality grows.

The geometric intuition breaks down in high dimensions. In 2D or 3D, most points are well-separated. In 50,000 dimensions, nearly all pairs of random vectors have approximately the same cosine similarity (close to zero), which means the similarity measure becomes uninformative for most pairs.

The solution is Latent Semantic Analysis (LSA), which applies Singular Value Decomposition (SVD) to the PMI-weighted co-occurrence matrix to compress it into a dense, low-dimensional representation (typically 100-300 dimensions). The SVD extracts the most important directions of variation in the matrix, discarding the noise and keeping the signal. Words that share similar semantic contexts end up close together in this compressed space, even if their raw co-occurrence profiles differed in detail.

The classical pipeline for distributional word representations works as follows. You start with a co-occurrence matrix built from raw counts, apply PMI weighting to correct for function word inflation, apply SVD to reduce dimensionality and remove noise, and end up with dense vector representations for words. This is the LSA pipeline, and it was the dominant approach to distributional word representations from the mid-1990s through approximately 2013.

When Word2Vec appeared in 2013 (Mikolov et al.), it seemed to bypass this pipeline entirely, learning dense word vectors directly from raw text through a neural network prediction objective. But subsequent analysis by Levy and Goldberg (2014) showed that Word2Vec's skip-gram model with negative sampling is implicitly performing a factorization of a shifted PMI matrix. The neural approach and the matrix factorization approach are mathematically related; Word2Vec is not doing something entirely different from LSA, it is doing something subtly different in a much more computationally efficient way.

This connection matters because it means the concepts you are learning in this chapter, the co-occurrence matrix, PMI weighting, and SVD, are not obsolete classical techniques. They are the foundational substrate from which all subsequent word representation methods are derived. Transformer-based contextual embeddings differ from static distributional representations, but the distributional hypothesis that motivates co-occurrence counting is still the conceptual foundation. The transformer learns which context to attend to dynamically, but the notion that context determines meaning is unchanged from Firth's original insight.