Co-occurrence Matrices › Weighting by Distance

A uniform window treats all co-occurrences equally regardless of distance. A word 5 positions away gets the same weight as a word 1 position away. This is simple, but it discards information: words adjacent to the target are usually more semantically informative than words further away.

Think about what the immediate neighbors of a word tell you versus what distant context tells you. The word immediately to the left of a noun is often a determiner, adjective, or possessive, and the word immediately to the right is often a verb or preposition. These tight grammatical constraints are highly specific to the target word. A word 5 positions away is more loosely associated: it is in the same sentence, likely on the same topic, but not necessarily in a tight grammatical relationship.

By giving equal weight to near and far context, a uniform window blurs the distinction between tight semantic relationships and loose topical associations. Distance weighting restores this distinction by making the contribution of each co-occurrence inversely proportional to how far away the context word was.

Distance weighting addresses this by assigning co-occurrences a weight inversely proportional to the distance between the two words. The most common scheme is harmonic weighting:

$$w(d) = \frac{1}{d}$$

where $d$ is the distance in tokens between the target word and the context word. So a word 1 position away contributes 1 to the co-occurrence count, a word 2 positions away contributes 0.5, a word 3 positions away contributes 0.33, and so on.

For a target word at position $t$ in a sentence, the weighted co-occurrence count with word $j$ is:

$$C_{ij} = \sum_{\substack{p \in \text{context}(t) \\ \text{word at } p = j}} \frac{1}{|t - p|}$$

where the sum runs over all positions $p$ within the window where word $j$ appears, and $|t - p|$ is the distance from the target to that occurrence.

The harmonic function $1/d$ has a specific property that makes it appealing: it assigns weight 1 to the nearest context, then diminishes smoothly. The weights for distances 1 through 5 are: 1.0, 0.5, 0.33, 0.25, 0.20. The first neighbor gets five times the weight of the fifth neighbor. This is not arbitrary: it reflects the empirical observation that the first few context words are typically the most informative about the target word's meaning.

Harmonic weighting is not the only option. Researchers have explored several alternatives.

Linear weighting uses weights that decrease linearly with distance: $w(d) = (k + 1 - d) / k$ for a window of size $k$. This gives a fixed weight of $k/k = 1$ to distance 1 and a weight of $1/k$ to distance $k$. Linear weighting is simpler to reason about than harmonic weighting and works well in practice.

Exponential weighting uses $w(d) = e^{-\alpha d}$ for some decay parameter $\alpha$. This drops off more steeply than harmonic weighting for small $\alpha$ and can be tuned to emphasize very tight context more aggressively.

Binary weighting (the uniform window) is the simplest: all context words get weight 1 regardless of distance. While it ignores distance information, it is the default in many implementations and often works well enough that the added complexity of distance weighting is not justified.

The choice of weighting scheme matters less than the choice of window size, but distance weighting consistently improves word vector quality in downstream tasks, at the cost of slightly more complex construction. GloVe, which we will discuss later in the book, implicitly captures distance weighting through its training objective: the GloVe loss function weights co-occurrence events, and words that appear very frequently near each other (often because they are grammatically adjacent) receive higher weight in training.