CBOW Model › Key Parameters

The behavior of CBOW is controlled by several important hyperparameters. Choosing these correctly has a significant impact on the quality of the resulting embeddings.

Embedding dimension ($d$): The dimensionality of the word vectors. Common values range from 50 to 300. Higher dimensions can capture more nuance but require more training data and memory. On a small corpus, high-dimensional embeddings tend to overfit: the model has more degrees of freedom than the data can constrain, and the extra dimensions fill with noise. On large corpora (billions of words), embeddings of 300 dimensions consistently outperform smaller sizes. The sweet spot depends on corpus size and downstream task requirements.

Window size ($m$): The number of words on each side of the center to include as context. Larger windows capture broader topical context because words that appear in the same general topic area will co-occur within the wider window even if they are not directly adjacent. Smaller windows capture tighter syntactic context: adjectives and their nouns, verbs and their subjects. For tasks that depend on syntactic similarity (parsing, part-of-speech tagging), smaller windows (1 or 2) tend to work better. For tasks that depend on semantic similarity (topic modeling, information retrieval), larger windows (5 or more) tend to be superior.

Learning rate: The step size for gradient descent updates. Too large causes divergence; too small causes slow convergence. Typically initialized around 0.025 and decayed during training. The original Word2Vec implementation uses a linear decay schedule: starting at 0.025 and linearly decreasing to 0.0001 as training progresses. This adaptive schedule allows aggressive initial learning while stabilizing the final embeddings.

Vocabulary size ($V$): Determined by the corpus. Larger vocabularies require more memory for the output matrix $\mathbf{W}'$ (size $d \times V$) and make the softmax sum more expensive to compute. With $V = 100{,}000$ and $d = 200$, the output matrix requires roughly 80 MB of memory. For practical applications with full softmax, vocabulary sizes above 100,000 become computationally prohibitive, which motivates the approximations covered in the next chapter.

Minimum count: Words appearing fewer than this many times in the corpus are excluded from the vocabulary. Setting min_count to 5 is common, filtering out rare words that have insufficient data to train well. These rare words receive so few gradient updates that their embeddings remain neededly random. This provides no useful information. Excluding them reduces vocabulary size, speeds up training, and avoids including low-quality embeddings in later applications.

The hyperparameter sensitivity plot illustrates how window size and embedding dimension affect the training trajectory. Wider windows and larger embedding dimensions both reduce the final training loss on this corpus, but they also slow each training epoch because wider windows produce more context combinations and larger matrices require more computation. In production, these tradeoffs must be balanced against available compute and the requirements of the downstream task.