Co-occurrence Matrices › Co-occurrence Matrices

If you count how often words appear near each other across a large corpus, you capture something surprisingly deep about language. Words that frequently share context tend to share meaning. "Bank" appears near "river" in some documents and near "loan" in others. "Doctor" appears near "patient," "diagnosis," and "hospital." "Python" appears near "code," "script," and "programming." These co-occurrence patterns, summarized in a matrix, form one of the oldest and most powerful representations in computational linguistics.

A co-occurrence matrix turns text into numbers without throwing away the relational structure between words. Unlike simple word counts, it records not what words appear, but what words appear together. That distinction matters enormously. Two synonyms like "begin" and "start" may have different raw frequencies, but they share nearly the same co-occurrence neighborhood, which is what makes them semantically equivalent in this framework. Frequency tells you how common a word is. Co-occurrence tells you what world a word inhabits.

This representational shift, from individual word frequencies to relational statistics between words, was one of the most important transitions in the history of computational linguistics. It transformed semantics from a hand-crafted ontology problem into a statistical inference problem. You no longer needed to program in the meaning of words. You could read their meaning off the data by examining who their neighbors were.

This chapter covers the two primary forms of co-occurrence matrices (word-word and word-document), the decisions that shape them (window size, distance weighting, directionality), the statistical properties they exhibit (sparsity, power-law distributions, dominance of function words), and the efficient algorithms used to construct them at scale. Understanding these matrices in depth matters because every major advance in distributional word representations, from PMI-weighted matrices to LSA to Word2Vec to transformer attention, can be understood as a refinement of the ideas here.