Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Learning Rate Warmup › Linear Warmup

Michael BrenndoerferMarch 24, 20267 min read

Language AI HandbookAll chapters

Part 32 of 67Training Optimization

Chapter 231Learning Rate Warmup › Linear Warmup

Reading section 5 of 16 · Linear Warmup · 7 minSection 5 of 16 · 7 min

Section 5 of 16 · 7 min read

Learning Rate Warmup › Linear WarmupOne section · Sections in this chapter5 / 16 ↓

Sections in this chapter16

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The most common warmup strategy is linear warmup, where the learning rate increases from zero (or a small initial value) to the target learning rate in a straight line over $T_{\text{warmup}}$ steps.

At training step $t$ , the learning rate during warmup is computed as a simple linear fraction of the peak learning rate:

\eta_t = \eta_{\max} \cdot \frac{t}{T_{\text{warmup}}}

where:

$\eta_t$ is the learning rate at step $t$
$\eta_{\max}$ is the target (peak) learning rate, the value you have tuned for stable training
$T_{\text{warmup}}$ is the total number of warmup steps (the duration of the warmup phase)
$t$ ranges from $1$ to $T_{\text{warmup}}$

At step $t = 1$ , the learning rate is $\eta_{\max} / T_{\text{warmup}}$ , which is nearly zero for typical warmup durations. At step $T_{\text{warmup}}$ , the fraction equals 1 and the learning rate reaches its peak value $\eta_{\max}$ .

After warmup completes (at $t > T_{\text{warmup}}$ ), the learning rate transitions to a decay schedule. If using cosine decay (covered in the next chapter), the combined schedule becomes:

\eta_t = \begin{cases} \eta_{\max} \cdot \dfrac{t}{T_{\text{warmup}}} & \text{if } t \leq T_{\text{warmup}} \\[8pt] \eta_{\min} + \dfrac{1}{2}(\eta_{\max} - \eta_{\min}) \left(1 + \cos\!\left(\pi \cdot \dfrac{t - T_{\text{warmup}}}{T_{\text{total}} - T_{\text{warmup}}}\right)\right) & \text{if } t > T_{\text{warmup}} \end{cases}

where:

$\eta_{\min}$ is the minimum learning rate at the end of training (often $0.1 \times \eta_{\max}$ )
$T_{\text{total}}$ is the total number of training steps across the entire training run

The cosine term $\cos(\pi \cdot \text{progress})$ evaluates to $+1$ at the start of the decay phase (progress $= 0$ ), giving $\eta_{\max}$ , and evaluates to $-1$ at the end (progress $= 1$ ), giving $\eta_{\min}$ . This creates a smooth, S-shaped transition from peak to minimum.

Out[3]:

Visualization

Line chart of learning rate vs training step showing linear warmup then cosine decay curve. — Learning rate schedule combining linear warmup and cosine decay over 100,000 training steps. The warmup phase (steps 0-2,000) ramps linearly from near-zero to the peak learning rate of 3e-4, then cosine decay smoothly reduces the rate to the minimum of 3e-5. The warmup phase occupies only 2% of total training while providing stability before the longer decay phase.

Why Linear and Not Exponential?Link Copied

You might wonder why linear warmup is preferred over exponential warmup, where the learning rate would grow multiplicatively. The answer comes down to how quickly the learning rate reaches its peak and how the optimizer benefits from spending time at intermediate learning rates.

Exponential warmup spends most of its budget at very low learning rates and reaches the peak only in the final few steps. Concretely, if you use a warmup factor of 10 and double the learning rate every step during warmup, the learning rate at the midpoint of warmup is only $\eta_{\max} / \sqrt{10} \approx 0.32 \cdot \eta_{\max}$ , but the Adam statistics accumulated during those slow early steps are dominated by tiny gradients that may not represent the true gradient scale at the peak learning rate. Linear warmup strikes a better balance, growing steadily so that the model is training at a reasonable pace throughout the warmup phase, and Adam's statistics are built up from a representative mix of learning rate levels.

Some practitioners use square-root warmup, where $\eta_t \propto \sqrt{t}$ , which is a middle ground between linear and exponential. The original Transformer paper "Attention Is All You Need" used an inverse-square-root schedule that starts high and then decays, which is mathematically equivalent to warmup followed by decay but expressed as a single formula. The schedule from the original paper combines the model dimension and step count:

\eta_t = d_{\text{model}}^{-0.5} \cdot \min\!\left(t^{-0.5},\; t \cdot T_{\text{warmup}}^{-1.5}\right)

where:

$d_{\text{model}}$ is the model's hidden dimension (a constant that scales the overall learning rate magnitude)
$t$ is the current step number (1-indexed)
$T_{\text{warmup}}$ is the warmup duration in steps

To understand this formula, note that the $\min$ selects the smaller of two expressions. For small $t$ (during warmup), the term $t \cdot T_{\text{warmup}}^{-1.5}$ is smaller, and the schedule grows linearly in $t$ . For large $t$ (after warmup), the term $t^{-0.5}$ becomes smaller, and the schedule decays as the inverse square root of $t$ . The crossover point between the two regimes occurs exactly at $t = T_{\text{warmup}}$ .

Despite its elegance, this schedule has been largely replaced by explicit linear warmup followed by cosine decay in modern practice, because the explicit version gives you direct control over the peak learning rate and the warmup duration independently. With the original Transformer schedule, changing $d_{\text{model}}$ changes the peak learning rate as a side effect, making it harder to tune.

Out[4]:

Visualization

Line chart comparing linear, square-root, and exponential warmup strategies over 2000 steps. — Comparison of three warmup strategies over 2,000 steps, all reaching the same peak learning rate. Linear warmup grows at a constant rate, spending roughly equal budget at each learning rate level. Square-root warmup starts faster but slows, and exponential warmup is extremely slow until the very end. Linear warmup provides the best balance between stability and training efficiency.

Line chart showing Adam initialization influence (beta2^t) decreasing over training steps, crossing the 1/e threshold near 1,000 steps. — Initialization influence on Adam's second-moment estimate over training steps with beta2=0.999. The weight of the zero-initialized state (beta2^t) decays from 1 toward 0, crossing the 1/e threshold (approx 0.37) at approximately 1,000 steps. Below this threshold, observed gradients dominate the estimate, making the second moment reliable and warmup safe to complete.