Word2Vec Training › Data Preprocessing Pipeline

Before training can begin, raw text must be transformed into a form the model can use. This preprocessing pipeline matters more than most practitioners realize. The decisions you make here, about tokenization, vocabulary size, and what to filter, directly affect embedding quality. Getting the preprocessing right is often the difference between embeddings that capture subtle semantic relationships and embeddings that are just noise in 300 dimensions.

Tokenization converts raw text into a sequence of tokens. For Word2Vec, simple whitespace tokenization often works surprisingly well. The key decision is what constitutes a token: do you lowercase everything, keep punctuation, split on hyphens? Each choice has downstream consequences.

Lowercasing is almost always the right call for Word2Vec. Without lowercasing, "Dog", "dog", and "DOG" become three separate vocabulary entries that receive independent gradient updates, even though they share essentially identical distributional context. On a billion-token corpus, "Dog" might appear rarely enough that it never develops a useful representation, while "dog" has thousands of training examples. Lowercasing merges these into a single entry and concentrates gradient signal.

Punctuation requires a separate decision. Keeping periods and commas as tokens means they become vocabulary entries, but they offer no semantic signal. Every sentence ends with a period regardless of content, so "period" would end up with an embedding that averages everything, similar to "the". Most practitioners strip punctuation entirely for Word2Vec.

Numbers present a different challenge. Replacing all numerals with a special <NUM> token prevents the vocabulary from being fragmented across "1", "2", ..., "10000", but it loses the ability to distinguish years from prices from distances. For most Word2Vec applications the tradeoff favors replacement.

Minimum frequency thresholding is perhaps the most consequential preprocessing decision. Rare words, those appearing fewer than 5 times in a billion-token corpus, cannot receive enough gradient updates to develop meaningful representations. Each training step moves an embedding a small amount; a word that appears only twice simply cannot travel far enough from its random initialization to reach a useful position in embedding space. Worse, rare words hurt other words: they each occupy a slot in the vocabulary, inflating the softmax denominator in naive implementations and adding noise to negative sampling. Removing them improves the embeddings of every word that remains.

A vocabulary size of 5 is small for this toy corpus, but in practice you would apply a minimum count threshold of 5 to 10 on large corpora. This removes rare words that would otherwise consume model parameters without receiving enough gradient updates to learn meaningful representations. On a web-scale corpus of 100 billion tokens, applying a minimum count of 5 typically reduces vocabulary size from tens of millions of unique strings down to a few million words with enough signal to learn from.

The choice of minimum count interacts with your embedding dimensionality. If you keep 100,000 vocabulary items and use 300-dimensional embeddings, your center embedding matrix alone is 30 million parameters. If you raise the minimum count to 100 and reduce vocabulary size to 50,000, you cut that in half while losing only very rare words that had poor representations anyway. Tighter frequency thresholds on smaller corpora and looser thresholds on larger corpora is a reasonable heuristic.

Given the processed token sequence, we generate (center, context) pairs using a sliding window. As described in the Skip-gram chapter, each word in the corpus becomes a center word, and all words within a window of size $w$ on either side become its context words.

The window size is a hyperparameter with meaningful semantic consequences, beyond ordinary parameter tuning. A small window of size 2 forces the model to focus on syntactic relationships: words that appear immediately adjacent tend to have grammatical dependencies, so the resulting embeddings encode part-of-speech and syntactic function. A large window of size 10 captures topical co-occurrence: words that appear anywhere in a ten-word span tend to share semantic domain, so the embeddings encode topic similarity. This is why researchers in the 2010s found that CBOW with a small window produced better syntactic embeddings while Skip-gram with a large window produced better semantic analogies.

There is also an important subtlety in how the original Word2Vec implementation handles window size: it samples a window size uniformly from 1 to $w$ for each center word, rather than using the same window size throughout. This means close neighbors contribute more often as context (they fall within any sampled window) while distant words contribute less often (they only fall within larger sampled windows). The effective weighting decreases with distance, which is a natural and desirable property: words right next to the center word are stronger context signals than words five positions away.