Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Word2Vec Training › Learning Rate Scheduling

Michael BrenndoerferApril 7, 20253 min read

Language AI HandbookAll chapters

Part 4 of 67Word Embeddings

Chapter 23Word2Vec Training › Learning Rate Scheduling

Reading section 5 of 14 · Learning Rate Scheduling · 3 minSection 5 of 14 · 3 min

Section 5 of 14 · 3 min read

Word2Vec Training › Learning Rate SchedulingOne section · Sections in this chapter5 / 14 ↓

Sections in this chapter14

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Word2Vec originally used stochastic gradient descent with a linearly decaying learning rate. The learning rate starts at an initial value $\alpha_0$ (typically 0.025 for Skip-gram) and decreases linearly to a minimum value $\alpha_{\min} = \alpha_0 / 10^4$ :

\alpha_t = \alpha_0 \cdot \left(1 - \frac{t}{T}\right) + \alpha_{\min} \cdot \frac{t}{T}

where:

$\alpha_t$ : learning rate at training step $t$
$\alpha_0$ : initial learning rate
$T$ : total number of training steps (total training pairs $\times$ epochs)
$t$ : current step count

This linear decay schedule has a specific purpose. Early in training, the embeddings start from random initialization and are far from useful values. A high learning rate allows large updates that move embeddings quickly toward meaningful regions of embedding space. The initial loss is high and the gradient surface is relatively smooth, so large steps are appropriate. As training progresses and embeddings settle into a good configuration, the loss surface becomes more irregular: nearby are both better and worse configurations. A lower learning rate allows fine-tuning without overshooting the optimal values.

Why linear decay specifically, rather than exponential decay or cosine annealing? The original choice was likely motivated by simplicity and the observation that Word2Vec training is already an approximation (negative sampling replaces the true softmax objective), so precise optimization schedules are less critical than for exact objectives. Empirically, linear decay from 0.025 to near zero works well and has become the standard for Word2Vec training.

The total steps $T$ depends on both the corpus size and the number of training epochs. For a corpus with $N$ tokens and window size $w$ , there are approximately $2wN$ training pairs per epoch. With 5 epochs, $T \approx 10wN$ . On a 1-billion-token corpus with window 5, that is roughly 50 billion steps, which is why Word2Vec training runs in hours rather than minutes even on modern hardware.

In[11]:

Code

def linear_lr_schedule(step, total_steps, lr_start=0.025, lr_min=2.5e-6):
    """Compute learning rate for a given step using linear decay."""
    progress = step / total_steps
    return lr_start * (1 - progress) + lr_min * progress


# Simulate a training run with 10,000 steps
total_steps = 10_000
lr_history = [
    linear_lr_schedule(step, total_steps) for step in range(total_steps)
]

Out[12]:

Visualization

Line plot of learning rate decreasing linearly from 0.025 to near zero across 10000 training steps. — Linear learning rate decay over 10,000 training steps as used in the original Word2Vec implementation. The rate begins at 0.025, enabling rapid movement from random initialization, then decays smoothly toward zero to allow fine-grained refinement as embeddings converge.

The linear schedule is not the only option. In practice, Gensim implements a similar schedule but recalculates the learning rate based on word tokens processed rather than gradient steps, which handles variable-length sentences more cleanly. Some practitioners use cosine annealing, which decays more slowly in the first half and more quickly in the second, giving more time at a moderate learning rate where the gradient signal is most useful. For Word2Vec specifically, the difference between decay schedules is relatively small because the model is training on a fixed dataset with no risk of catastrophic overfitting.