CBOW Model › Dynamic Window Sizes

Another practical technique from the original implementation is dynamic window sizing. Rather than always using the maximum window size $m$, the model samples the actual window size uniformly from $\{1, 2, \ldots, m\}$ for each training example.

This produces an interesting statistical effect. Words immediately adjacent to the center word are included in every window size from 1 to $m$, while words at the maximum distance are only included when the window size is sampled as $m$. Words at distance $k$ are included with probability $(m - k + 1) / m$.

The result is a soft weighting: nearby words contribute more frequently to training examples than distant words, without requiring any explicit weighting scheme. This implicitly encodes the linguistic intuition that a word's immediate neighbors carry more information about its meaning than words several positions away. Dynamic window sizing is enabled by default in the original Word2Vec implementation and is one reason the original CBOW embeddings outperformed naive implementations that used fixed window sizes.