CBOW Model › Training CBOW with Gradient Descent
Language AI HandbookAll chapters CBOW Model › Training CBOW with Gradient DescentOne section · Sections in this chapter7 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
With the forward pass and gradients in hand, we can train CBOW using stochastic gradient descent. Each training step processes one (context, target) pair, computes the loss, backpropagates, and updates both weight matrices.
Stochastic gradient descent updates parameters after each individual training example rather than accumulating gradients over the entire dataset. For word embedding training, this is essential: the dataset is too large to load into memory, and the stochastic updates provide a stream of small gradient signals that collectively shape the embedding space. Each update is noisy (a single training example is an imperfect estimate of the true gradient), but across millions of examples the noise averages out and the embeddings converge.
The learning rate controls the magnitude of each update. Starting with a relatively high learning rate and decaying it as training progresses is standard practice in Word2Vec. Early in training, large updates quickly move embeddings from their random initializations into meaningful regions. Later in training, smaller updates fine-tune the embeddings without overshooting.
def train_cbow(
corpus_words,
vocab,
word_to_idx,
embedding_dim=8,
window_size=2,
learning_rate=0.01,
epochs=200,
):
"""
Train a CBOW model using stochastic gradient descent.
Returns the trained embedding matrix.
"""
V = len(vocab)
np.random.seed(42)
W_emb = np.random.randn(V, embedding_dim) * 0.01
W_out = np.random.randn(embedding_dim, V) * 0.01
# Generate all training pairs
pairs = generate_cbow_pairs(corpus_words, window_size)
losses = []
for epoch in range(epochs):
epoch_loss = 0.0
np.random.shuffle(pairs)
for context_words, target_word in pairs:
# Forward pass
h_bar, probs = cbow_forward(
context_words, W_emb, W_out, word_to_idx
)
loss = compute_loss(target_word, probs, word_to_idx)
epoch_loss += loss
# Backward pass
dW, dW_prime = cbow_backward(
context_words,
target_word,
h_bar,
probs,
W_emb,
W_out,
word_to_idx,
)
# Update parameters
W_emb -= learning_rate * dW
W_out -= learning_rate * dW_prime
losses.append(epoch_loss / len(pairs))
return W_emb, W_out, losses
# Training corpus (repeated for more data)
corpus = (
" ".join(
[
"the quick brown fox jumps over the lazy dog",
"the dog chased the fox",
"the fox and the dog played",
"quick fox jumped high",
"lazy dog slept all day",
"the brown dog and the quick fox",
]
)
* 3
)
corpus_words = corpus.lower().split()
# Build vocabulary from corpus
full_vocab = sorted(set(corpus_words))
full_word_to_idx = {w: i for i, w in enumerate(full_vocab)}
# Train the model
W_trained, W_out_trained, train_losses = train_cbow(
corpus_words,
full_vocab,
full_word_to_idx,
embedding_dim=8,
window_size=2,
learning_rate=0.05,
epochs=300,
)Training complete. Vocabulary size: 17 Training pairs: 106 Epochs: 300 Final avg loss: 0.0536 Initial avg loss: 2.8332 Loss reduction: 98.1%
The loss curve shows the typical pattern for Word2Vec training: rapid early improvement as the model captures the most common patterns, followed by slower refinement. On small corpora the final loss is still relatively high because the model cannot distinguish many fine-grained contexts with limited data. Training on a real corpus of millions or billions of tokens would continue pushing the loss lower, and the embeddings would become much more precisely positioned.
Initialization and Stability
A practical note on initialization: CBOW parameters should be initialized with small random values, not zeros. If all embeddings start at zero, the gradient for every context word is identical (they all produce the same averaged vector), and the symmetry is never broken. Small random values ensure that different words receive different initial gradients, allowing them to diverge into distinct regions of the embedding space.
The output matrix is typically initialized with the same small-random scheme. Some implementations initialize to zeros, relying on the non-zero embeddings to break symmetry through the backward pass. Both approaches work in practice, though small random initialization for both matrices tends to converge slightly faster in our experience.