The Distributional Hypothesis › Context Window Size: What Gets Captured

Michael BrenndoerferMarch 29, 20252 min read
Language AI HandbookAll chapters
Part 3 of 67Distributional Semantics
Chapter 15The Distributional Hypothesis › Context Window Size: What Gets Captured
Reading section 9 of 16 · Context Window Size: What Gets Captured · 2 minSection 9 of 16 · 2 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 15 of 513

Section 9 of 16 · 2 min read

The Distributional Hypothesis › Context Window Size: What Gets CapturedOne section · Sections in this chapter9 / 16 ↓
Sections in this chapter16
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The choice of window size is a model hyperparameter that affects what kind of similarity the system learns. Let's visualize how similarity estimates change as we vary window size.

Out[14]:
Visualization
Line chart of cosine similarity vs context window size for five word pairs.
Cosine similarity between word pairs across context window sizes from 1 to 6. Semantically similar pairs (cat/dog, run/walk, apple/banana) maintain high similarity across all window sizes, while unrelated pairs (cat/apple, run/banana) remain near zero, confirming that distributional signal is robust to the choice of window size.

Semantically related pairs like "cat"/"dog" and "run"/"walk" generally score high across all window sizes because their shared contexts appear at both close and distant offsets. Unrelated pairs like "cat"/"apple" and "run"/"banana" remain low. The window size primarily affects how much overlap accumulates in the middle range, not which pairs are fundamentally similar.

This stability across window sizes in a small corpus is partly due to our corpus design: we deliberately wrote parallel sentences. In real corpora, the relationship between window size and captured similarity type is more pronounced, and choosing the right window size for your task matters.