Co-occurrence Matrices › A Worked Example

Michael BrenndoerferUpdated January 25, 20262 min read
Language AI HandbookAll chapters
Part 3 of 67Distributional Semantics
Chapter 16Co-occurrence Matrices › A Worked Example
Reading section 8 of 13 · A Worked Example · 2 minSection 8 of 13 · 2 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 16 of 513

Section 8 of 13 · 2 min read

Co-occurrence Matrices › A Worked ExampleOne section · Sections in this chapter8 / 13 ↓
Sections in this chapter13
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's trace through a small example to make the construction concrete. Consider this three-sentence corpus with a window of size 2:

  1. "the cat sat on the mat"
  2. "the cat ate the rat"
  3. "the dog sat on the log"

The vocabulary (excluding stopwords for clarity, though real implementations usually include them) is: {cat, sat, mat, ate, rat, dog, log, on}.

For the word "sat" with window k=2k = 2, the contexts are:

  • In sentence 1: "cat" (distance 1 left), "on" (distance 1 right), "the" (distance 2 right)
  • In sentence 3: "dog" (distance 1 left), "on" (distance 1 right), "the" (distance 2 right)

So the row for "sat" in the co-occurrence matrix will have counts: cat: 1, on: 2, dog: 1, and "the": 2 (within the vocabulary).

With harmonic distance weighting: cat gets weight 1.0 (distance 1), on gets 1.0 (distance 1), dog gets 1.0 (distance 1). For "the" at distance 2, it gets 0.5 in each sentence, total 1.0.

Now let's trace the row for "cat." In sentence 1, "cat" appears at position 1 (0-indexed). Context words within window 2 are: "the" at distance 1 (left), "sat" at distance 1 (right), "on" at distance 2 (right). In sentence 2, "cat" appears at position 1 again. Context words: "the" at distance 1 (left), "ate" at distance 1 (right), "the" at distance 2 (right).

Summing across sentences:

  • "the": appears at distance 1 in sentences 1 and 2 (uniform count: 2), and at distance 2 in sentence 2 (uniform count: 1), giving uniform total 3.
  • "sat": appears at distance 1 in sentence 1 (uniform count: 1).
  • "ate": appears at distance 1 in sentence 2 (uniform count: 1).
  • "on": appears at distance 2 in sentence 1 (uniform count: 1).

With harmonic weighting, the count for "the" would be 1.0+1.0+0.5=2.51.0 + 1.0 + 0.5 = 2.5, compared to the uniform count of 3. The relative reduction for "the" is smaller here, but in a large corpus where "the" appears at all distances from many words, harmonic weighting substantially reduces its dominance.

The full matrix construction follows this same process for every token in the corpus, row by row, for every word in the vocabulary.