Co-occurrence Matrices › Limitations and Impact

Co-occurrence matrices made distributional semantics computationally tractable for the first time, but they also introduced several limitations that shaped subsequent research.

The most pressing limitation is scalability. Even with sparse storage, a $V \times V$ matrix for a vocabulary of 500,000 words requires billions of nonzero entries to be handled. The construction itself requires a full pass through the corpus for every vocabulary size configuration, and modifying the window size or weighting scheme requires rebuilding the matrix from scratch. For research applications that need to explore many configurations, this computational cost adds up.

The matrix is also static: once built, it does not update as new text arrives. In a production search system that needs to incorporate new documents continuously, a static co-occurrence matrix becomes outdated and requires periodic full rebuilds. This is one of the practical motivations for moving toward online learning methods like Word2Vec, which can be updated incrementally.

The raw counts are also poor representations on their own: function words like "the," "of," and "in" co-occur with almost every content word, creating massive count imbalances that flood the signal from meaningful associations. This motivated post-processing with TF-IDF weighting in the document context, and with PMI (Pointwise Mutual Information) in the word-word context. We cover PMI in the next chapter.

The dominance of function words is a symptom of a deeper issue about what the raw matrix is measuring. Co-occurrence counts conflate two distinct types of association: grammatical co-occurrence (words that appear together because of syntactic constraints, like "the" appearing before nouns) and semantic co-occurrence (words that appear together because they refer to related things). Function words primarily create grammatical co-occurrences, while content words primarily create semantic co-occurrences. PMI weighting cleanly separates these two types by normalizing for each word's marginal frequency.

A deeper limitation is dimensionality. A word's representation is a vector of length $V$, often 50,000 or more dimensions. Most of these dimensions are zero or nearly zero. Operations like cosine similarity are computationally expensive in such high-dimensional sparse spaces, and the curse of dimensionality means that cosine distances become increasingly uninformative as dimensionality grows. This limitation motivated Latent Semantic Analysis (LSA), which applies Singular Value Decomposition to compress co-occurrence matrices into dense, low-dimensional representations. LSA is covered in the SVD chapter.

Another fundamental limitation of co-occurrence matrices, one that persists even after PMI weighting and SVD compression, is their inability to handle polysemy. Every word gets a single vector, regardless of how many senses it has. "Bank" in the financial sense and "bank" in the river sense collapse into a single point in the vector space. The vector ends up in an awkward intermediate position that is not fully representative of either sense.

This context blindness extends beyond polysemy to all forms of contextual variation. The meaning of "good" in "a good actor" and "a good person" is subtly different, but a static co-occurrence matrix assigns both usages to the same vector. This motivated the shift toward contextual word embeddings, where the representation of a word depends on its specific context. ELMo (2018) and BERT (2018) are the first systems to produce such contextual representations at scale, and the transformer architecture that enables them can be understood as a learned, dynamic context weighting mechanism. Instead of a fixed window, the transformer learns to weight context positions differently for each occurrence of a word, effectively learning a different vector for each context.

Despite these limitations, the impact of co-occurrence matrices was foundational. They provided the first systematic way to compute word similarity from distributional evidence, without any hand-crafted rules or semantic ontologies. The computational linguists of the 1990s and 2000s used these matrices to demonstrate that meaningful semantic relationships like synonymy, antonymy, and topical relatedness could be discovered automatically from text statistics. This was a conceptual breakthrough that opened the door to Word2Vec, GloVe, and eventually transformer-based representations.

The question "can computers understand meaning?" had always been answered with skepticism or appeals to philosophical difficulty. Co-occurrence matrices gave a concrete, empirically grounded answer: not full understanding, but a measurable approximation of semantic similarity that correlates strongly with human judgments. When you showed researchers in 1995 that a co-occurrence matrix could correctly identify "surgeon" as more similar to "physician" than to "carpenter," without any built-in knowledge of medicine, it changed how the field thought about the relationship between text statistics and meaning.

Co-occurrence matrices also made explicit the fundamental tradeoff that persists throughout distributional semantics: the choice of context definition determines what kind of similarity the representation captures. This insight, that there is no single correct definition of "context," continues to influence how we design contextual representations in modern language models. Transformer attention, in some sense, is a learned context weighting mechanism: instead of a fixed window, the model learns to weight context positions dynamically based on content. The co-occurrence matrix is not a primitive predecessor of the transformer; it is the same idea expressed more simply.