Cosine Learning Rate Schedule › The Cosine Decay Formula

The cosine schedule derives its shape from the cosine function, which decreases smoothly from 1 to $-1$ over a half-period. By remapping this interval to span from the initial learning rate down to a minimum value, we get a decay profile that is gentle near the boundaries and steeper in the middle.

We want a function that starts at $\eta_{\max}$ when $t = 0$ and ends at $\eta_{\min}$ when $t = T$, following a smooth cosine curve in between. The standard cosine decay formula achieves this as follows:

where:

• $\eta(t)$: the learning rate at step $t$
• $\eta_{\max}$: the peak learning rate at the start of the schedule (after any warmup period)
• $\eta_{\min}$: the floor learning rate, often set to 0 or a small fraction of $\eta_{\max}$ such as $10^{-5}$
• $t$: the current step index, ranging from 0 to $T$
• $T$: the total number of decay steps
• $\cos(\pi t / T)$: the cosine term that drives the smooth interpolation from 1 (at $t=0$) to $-1$ (at $t=T$)

The expression $\frac{1}{2}(1 + \cos(\cdot))$ is sometimes called the "raised cosine" because it shifts and scales the cosine from its natural range $[-1, 1]$ up into the range $[0, 1]$. Multiplying by $(\eta_{\max} - \eta_{\min})$ stretches this unit interval to the desired learning rate range, and adding $\eta_{\min}$ lifts the floor.

To see why this particular construction works, think of it as a two-step transformation. The raw cosine $\cos(\pi t / T)$ takes values in $[-1, 1]$: it equals $+1$ at $t = 0$, passes through $0$ at $t = T/2$, and reaches $-1$ at $t = T$. We want a multiplier that goes from 1 to 0 (not from 1 to $-1$), so we add 1 and divide by 2 to get $\frac{1 + \cos(\pi t / T)}{2}$. This quantity ranges from 1 down to 0. Multiplying by $(\eta_{\max} - \eta_{\min})$ scales that unit range to match the desired learning rate range, and adding $\eta_{\min}$ ensures we never go below the floor.

Verification at boundary values. Let us confirm the formula produces the right values at $t = 0$ and $t = T$:

At $t = 0$:

At $t = T$:

The schedule begins at $\eta_{\max}$ and ends exactly at $\eta_{\min}$, as intended. At the midpoint $t = T/2$, the cosine evaluates to $\cos(\pi/2) = 0$, giving a learning rate of exactly the average $(\eta_{\max} + \eta_{\min}) / 2$.

One of the most instructive ways to understand the cosine schedule is to look at the derivative of $\eta(t)$ with respect to $t$. The derivative tells us how fast the learning rate is falling at each step:

The sine function starts at zero, rises to its maximum at $t = T/2$, and falls back to zero at $t = T$. This means the learning rate is decreasing most rapidly at the midpoint of training, and barely changing at all at the very beginning and very end. Compare this with linear decay, where the derivative is a constant $-(\eta_{\max} - \eta_{\min}) / T$ at every step: the rate of decrease is the same regardless of how far along training has progressed.

This derivative structure is exactly what we want. At the start of training, gradients are large and noisy, and a slowly-changing learning rate gives the optimizer time to orient itself without the additional disruption of a rapidly shifting step size. Near the end of training, the learning rate itself is already very small, and a nearly flat derivative means the optimizer is making barely-perceptible adjustments to the rate, allowing for very fine-grained convergence. The middle phase, where the learning rate is most actively shrinking, corresponds to the period when the optimizer is making the transition from broad exploration to local refinement.

Step decay schedules drop the learning rate by a fixed factor at discrete checkpoints. This creates a jarring discontinuity each time the rate drops: the optimizer is cruising along at one scale and suddenly the effective step size contracts by an order of magnitude. In practice, these drops often coincide with sudden sharp decreases in training loss followed by a plateau at the new rate. While this works, it makes scheduling sensitive to the choice of when to drop.

After each step-decay drop, the loss tends to spike briefly as the optimizer adjusts to the new scale, then quickly drops to a lower floor than before. This "shock and reset" behavior works, but it suggests the model is doing a kind of forced convergence at each checkpoint rather than settling smoothly. The timing of drops has to be chosen carefully: too early and the model has not finished benefiting from the higher rate; too late and it has been oscillating around a minimum without making progress.

Linear decay, covered in the Learning Rate Decay chapter, avoids this problem with a straight-line decrease. However, linear decay is fastest right at the start, when the model may still be in a phase where slightly larger steps would help exploration, and it decreases at a uniform rate that ignores the characteristic phases of learning.

Cosine decay allocates decay budget differently. The rate decreases slowly at first, giving the model time to make larger initial progress. It decelerates more rapidly through the middle of training when parameters are converging broadly. Near the end, the rate again slows its descent, allowing the model to settle delicately into a local minimum without overshooting. This pacing matches the natural phases of gradient descent quite well.

There is also a practical advantage: cosine decay requires no manual checkpoint selection. With step decay, you choose specific milestones, and if they are wrong for your architecture, problem, or batch size, performance suffers. The cosine schedule adapts its budget continuously and automatically based solely on the current step fraction $t/T$.

To make the formula concrete, let us trace through the exact values at several checkpoints for a schedule with $\eta_{\max} = 3 \times 10^{-4}$, $\eta_{\min} = 3 \times 10^{-5}$, and $T = 1000$ steps. These values are typical for training a medium-sized language model with AdamW.

At $t = 0$: $\cos(0) = 1$, so $\eta = 3 \times 10^{-5} + \frac{1}{2}(2.7 \times 10^{-4})(2) = 3 \times 10^{-4}$. This is the peak, as expected.

At $t = 250$ (25% through training): $\cos(\pi \cdot 250 / 1000) = \cos(\pi/4) \approx 0.707$. So:

The learning rate has decayed from $3 \times 10^{-4}$ to approximately $2.60 \times 10^{-4}$, a reduction of only about 13% despite being 25% through training. This is the "slow start" effect at work.

At $t = 500$ (midpoint): $\cos(\pi/2) = 0$, so $\eta = 3\times10^{-5} + \frac{1}{2}(2.7\times10^{-4})(1) = 1.65\times10^{-4}$. This is exactly the midpoint between $\eta_{\max}$ and $\eta_{\min}$.

At $t = 750$ (75% through training): $\cos(3\pi/4) \approx -0.707$. So:

By 75% of training, the learning rate has fallen to about $6.96 \times 10^{-5}$, already very close to the minimum of $3 \times 10^{-5}$. The remaining 25% of training will use a very small, slowly-changing learning rate, which is ideal for final refinement.

At $t = 1000$: $\cos(\pi) = -1$, so $\eta = 3\times10^{-5} + 0 = 3\times10^{-5} = \eta_{\min}$.

This worked example illustrates the curve's pacing clearly: the cosine schedule changes slowly near its peak, drops most rapidly around the midpoint, and slows again near the floor. A linear schedule also reaches the midpoint value of $1.65\times10^{-4}$ at step 500, but it reaches $2.60\times10^{-4}$ at about step 148 rather than step 250. Cosine therefore spends roughly 70% more steps above $2.60\times10^{-4}$ in this example; both schedules still spend half of training above the midpoint value.