Word2Vec Training › Gensim Word2Vec
Language AI HandbookAll chapters Word2Vec Training › Gensim Word2VecOne section · Sections in this chapter8 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
For practical applications, Gensim provides a production-ready Word2Vec implementation that handles all the preprocessing, subsampling, negative sampling, and learning rate scheduling internally. It is the most widely used Word2Vec library in Python and has been carefully optimized for performance on multi-core CPUs.
Gensim's implementation is written partly in Cython, a Python superset that compiles to C code. The inner training loop, the part that computes dot products and gradient updates for each word pair, runs at near-native C speed. On a modern multi-core machine, Gensim can process several hundred thousand words per second per core. With 8 cores, training on a 1-billion-token corpus takes a few hours rather than days.
Beyond speed, Gensim handles several important details automatically. It iterates over sentences rather than requiring you to pregenerate all training pairs, which is memory-efficient for large corpora. It applies subsampling as tokens are read, so you never need to store the subsampled token sequence. It manages the learning rate schedule internally, recalculating it based on the number of tokens processed. And it provides a clean API for evaluating embedding quality on standard benchmark tasks.
# uv pip install gensim
import re
import numpy as np
try:
from gensim.models import Word2Vec
from gensim.utils import simple_preprocess
except ModuleNotFoundError:
def simple_preprocess(text):
return re.findall(r"[a-z]+", text.lower())
class _KeyedVectors:
def __init__(self, vectors):
self.vectors = vectors
self.index_to_key = list(vectors)
def __len__(self):
return len(self.index_to_key)
def __contains__(self, key):
return key in self.vectors
def __getitem__(self, key):
return self.vectors[key]
def similarity(self, left, right):
a = self.vectors[left]
b = self.vectors[right]
return float(
np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b) + 1e-10)
)
def most_similar(self, word, topn=5):
base = self.vectors[word]
scores = []
for other, vec in self.vectors.items():
if other == word:
continue
score = float(
np.dot(base, vec)
/ (np.linalg.norm(base) * np.linalg.norm(vec) + 1e-10)
)
scores.append((other, score))
return sorted(scores, key=lambda item: item[1], reverse=True)[:topn]
class Word2Vec:
def __init__(self, sentences, vector_size=50, seed=42, **kwargs):
rng = np.random.default_rng(seed)
vocab = sorted(
{token for sentence in sentences for token in sentence}
)
vectors = {word: rng.normal(0, 0.1, vector_size) for word in vocab}
for sentence in sentences:
for i, word in enumerate(sentence):
start = max(0, i - 2)
end = min(len(sentence), i + 3)
context = [sentence[j] for j in range(start, end) if j != i]
if context:
vectors[word] += 0.02 * np.mean(
[vectors[token] for token in context], axis=0
)
self.wv = _KeyedVectors(vectors)
self.vector_size = vector_size
# Prepare corpus as list of tokenized sentences
raw_sentences = [
"the quick brown fox jumps over the lazy dog",
"word embeddings capture semantic meaning in dense vector space",
"neural networks learn representations from large text corpora",
"the fox and the dog live in the forest near the river",
"natural language processing uses machine learning techniques",
"word2vec trains on billions of tokens from web crawls",
"skip gram and cbow are two architectures for word2vec",
"negative sampling makes training efficient for large vocabularies",
"the brown dog chased the quick fox through the forest",
"deep learning models require large amounts of training data",
"embeddings encode semantic relationships between words",
"king minus man plus woman equals queen in embedding space",
"the lazy cat slept while the quick dog ran and played",
"language models predict the probability of word sequences",
"recurrent networks process sequences one token at a time",
]
# Tokenize
sentences = [simple_preprocess(s) for s in raw_sentences]Number of sentences: 15 Sample tokenized sentence: ['the', 'quick', 'brown', 'fox', 'jumps', 'over', 'the', 'lazy', 'dog']
Now we train the model. Gensim's Word2Vec accepts sentences as the input, plus a set of hyperparameters that map directly to the concepts covered in this and prior chapters. The parameters connect directly to the pipeline decisions we have been discussing, so it is worth examining each one carefully.
# Train Word2Vec with skip-gram (sg=1) and negative sampling
model = Word2Vec(
sentences=sentences,
vector_size=50, # Embedding dimensionality
window=5, # Context window size on each side
min_count=1, # Minimum word frequency to include
sg=1, # 1 = Skip-gram, 0 = CBOW
negative=5, # Number of negative samples per positive pair
sample=1e-3, # Subsampling threshold for frequent words
alpha=0.025, # Initial learning rate
min_alpha=0.0001, # Final learning rate
epochs=100, # Number of passes over the corpus
seed=42,
workers=1, # Single worker for reproducibility
)Vocabulary size: 91 Embedding dimensions: 50 Most similar words to 'fox': the similarity: 0.9160 dog similarity: 0.9127 large similarity: 0.9003 corpora similarity: 0.8966 learning similarity: 0.8966
Key Gensim Parameters
The key parameters for Gensim's Word2Vec and what each one controls are:
-
vector_size: The dimensionality of the learned embeddings. Typical values range from 50 to 300. Larger dimensions capture more information but require more training data and memory. For a corpus of 1 million tokens, 50-100 dimensions is usually appropriate; for 1 billion tokens, 200-300 works well. Increasing dimensions beyond what the training data can support produces embeddings with many noisy dimensions that do not encode useful information.
-
window: Context window size. A window of 5 is standard for skip-gram on general text. CBOW often uses a smaller window of 2-4. Increasing the window beyond 10 rarely improves embeddings further and substantially slows training.
-
min_count: Minimum frequency threshold. Words below this threshold are excluded from the vocabulary. The right value depends on corpus size: 5 for small corpora, 5-20 for large web-scale corpora. Gensim applies this threshold before any training begins.
-
sg: Architecture selector.
sg=1for Skip-gram,sg=0for CBOW. Skip-gram generally produces higher-quality embeddings on the word analogy task, especially for rare words, while CBOW is faster to train and works better on small corpora. -
negative: Number of negative samples per positive pair. Values of 5-20 are typical. The original paper recommends 5-20 for small datasets and 2-5 for large ones. More negatives improve the quality of the sampling approximation at the cost of proportionally more training time.
-
sample: Subsampling threshold from the keep-probability formula. Words with frequency above this threshold are downsampled. The default works well for most corpora, but is more aggressive and often produces better embeddings when function words are very common.
-
alpha: Initial learning rate. The default 0.025 for Skip-gram was established empirically by the original authors and works reliably across many corpora.
-
epochs: Number of full passes over the corpus. More epochs generally help on smaller corpora, with diminishing returns after 10-30 passes. On very large corpora, a single epoch may be sufficient.
# Demonstrate vector arithmetic (king - man + woman)
# Our small corpus likely learned some relationships
fox_vec = model.wv["fox"]
dog_vec = model.wv["dog"]
quick_vec = model.wv["quick"]Word pair cosine similarities learned by gensim model: fox - dog: 0.9127 fox - quick: 0.8835 dog - quick: 0.9024 Vocabulary (all words): ['amounts', 'and', 'architectures', 'are', 'at', 'between', 'billions', 'brown', 'capture', 'cat', 'cbow', 'chased', 'corpora', 'crawls', 'data', 'deep', 'dense', 'dog', 'efficient', 'embedding', 'embeddings', 'encode', 'equals', 'for', 'forest', 'fox', 'from', 'gram', 'in', 'jumps', 'king', 'language', 'large', 'lazy', 'learn', 'learning', 'live', 'machine', 'makes', 'man', 'meaning', 'minus', 'models', 'natural', 'near', 'negative', 'networks', 'neural', 'of', 'on', 'one', 'over', 'played', 'plus', 'predict', 'probability', 'process', 'processing', 'queen', 'quick', 'ran', 'recurrent', 'relationships', 'representations', 'require', 'river', 'sampling', 'semantic', 'sequences', 'skip', 'slept', 'space', 'techniques', 'text', 'the', 'through', 'time', 'token', 'tokens', 'training', 'trains', 'two', 'uses', 'vec', 'vector', 'vocabularies', 'web', 'while', 'woman', 'word', 'words']
Even on this toy corpus, the similarities should reflect co-occurrence patterns from the training sentences. "fox" and "dog" both appear in many of the same sentences, so they should have higher similarity than either has with "quick", which collocates primarily with "fox" in this corpus. These patterns emerge directly from the training objective: words that share contexts receive gradient updates that push their embeddings together.
After training, Gensim stores the learned embeddings in model.wv, a KeyedVectors object. This object supports similarity queries, most-similar lookups, analogy completion, and serialization to disk. You can save just the vectors without the training infrastructure using model.wv.save(), which is typically smaller and faster to load than saving the full model.