Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Learning Rate Decay › Cosine Annealing

Michael BrenndoerferJanuary 24, 20266 min read

Language AI HandbookAll chapters

Part 32 of 67Training Optimization

Chapter 232Learning Rate Decay › Cosine Annealing

Reading section 6 of 15 · Cosine Annealing · 6 minSection 6 of 15 · 6 min

Section 6 of 15 · 6 min read

Learning Rate Decay › Cosine AnnealingOne section · Sections in this chapter6 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Cosine annealing uses a half-cosine curve to reduce the learning rate from a maximum value to a minimum value over a fixed number of steps. It became widely adopted after Loshchilov and Hutter introduced it in 2017 and combined it with warm restarts to create SGDR (Stochastic Gradient Descent with Warm Restarts).

Why Cosine?Link Copied

The choice of a cosine curve rather than a simple straight line or exponential is motivated by the shape of the decay, not just convenience. At the start of the cosine curve (near $t = 0$ ), the cosine function changes slowly, meaning the learning rate stays near its maximum for a while and decreases gently at first. Near the end (as $t$ approaches $T_{\max}$ ), the cosine function is near $-1$ and also changing slowly, meaning the learning rate settles near its minimum and finishes smoothly. In the middle, the cosine passes through its steepest section, driving the sharpest decline.

This shape is intuitively appealing: the model starts at full speed, gradually shifts into a slower and slower rate, and comes to a gentle stop. Contrast this with linear decay, which reduces the learning rate at a constant rate regardless of where training is, and step decay, which drops abruptly. Cosine annealing provides a natural deceleration curve.

The cosine also has a nice mathematical property: the derivative of the cosine schedule is zero at both endpoints. This means the learning rate changes most slowly exactly at the start and end of the schedule, which is often where training is most sensitive to abrupt changes.

FormulationLink Copied

The basic cosine annealing schedule is:

\eta_t = \eta_{\min} + \frac{1}{2}\left(\eta_{\max} - \eta_{\min}\right)\!\left(1 + \cos\!\left(\frac{\pi t}{T_{\max}}\right)\right)

where:

$\eta_{\min}$ : the minimum learning rate (often 0 or a small positive value like $10^{-5}$ )
$\eta_{\max}$ : the maximum learning rate
$T_{\max}$ : the number of steps for one half-cosine period
$t$ : the current step within the period

At $t = 0$ , the cosine term is $\cos(0) = 1$ , giving $\eta_0 = \eta_{\max}$ . At $t = T_{\max}$ , the cosine term is $\cos(\pi) = -1$ , giving $\eta_{T_{\max}} = \eta_{\min}$ . In between, the rate follows the smooth cosine curve. The factor of $1/2$ ensures the output lies in the range $[\eta_{\min}, \eta_{\max}]$ .

Worked Numerical ExampleLink Copied

Let us trace through a specific case to build intuition. Suppose $\eta_{\max} = 0.1$ , $\eta_{\min} = 0.001$ , and $T_{\max} = 100$ .

At $t = 0$ :

\eta_0 = 0.001 + \frac{1}{2}(0.1 - 0.001)(1 + \cos(0)) = 0.001 + \frac{1}{2}(0.099)(2) = 0.1

At $t = 25$ (one quarter through):

\eta_{25} = 0.001 + \frac{1}{2}(0.099)\left(1 + \cos\!\left(\frac{\pi \cdot 25}{100}\right)\right) = 0.001 + \frac{1}{2}(0.099)(1 + \cos(0.25\pi))

Since $\cos(0.25\pi) = \cos(45°) \approx 0.707$ , this gives approximately $0.001 + 0.0495 \times 1.707 \approx 0.086$ .

At $t = 50$ (halfway):

\eta_{50} = 0.001 + \frac{1}{2}(0.099)(1 + \cos(0.5\pi)) = 0.001 + \frac{1}{2}(0.099)(1 + 0) = 0.001 + 0.0495 \approx 0.0505

At $t = 100$ (end):

\eta_{100} = 0.001 + \frac{1}{2}(0.099)(1 + \cos(\pi)) = 0.001 + \frac{1}{2}(0.099)(0) = 0.001

Halfway through training, the learning rate is approximately half of the initial value. The curve is symmetric around the halfway point in a certain sense: the first quarter (epochs 0-25) sees a relatively small drop from 0.1 to 0.086, while the third quarter (epochs 50-75) sees a larger drop from 0.0505 to about 0.016. The decline accelerates in the middle of the schedule.

Warm RestartsLink Copied

The standard cosine schedule ends at $\eta_{\min}$ and stays there. Loshchilov and Hutter proposed restarting the schedule after reaching the minimum, cycling back to $\eta_{\max}$ and annealing again. This is cosine annealing with warm restarts, sometimes written as SGDR or CosineAnnealingWarmRestarts.

The restart strategy has an appealing justification: near the end of each cycle, the low learning rate drives the model into a local minimum. The restart then kicks the model out of that minimum with a higher learning rate, potentially allowing it to find a flatter, more generalizable minimum in the next cycle. The intuition is that flat minima generalize better than sharp minima, and warm restarts encourage exploration of the loss landscape.

Restarts can also be paired with a multiplicative increase in the cycle length. Each successive cycle is made longer by a factor $T_{\text{mult}}$ , so the model spends more and more time at low learning rates as training progresses. A typical choice is $T_{\text{mult}} = 2$ , doubling the cycle length each restart.

Cosine Annealing with Warm Restarts

A learning rate schedule that follows a cosine curve from maximum to minimum, then resets to the maximum and repeats. The restart allows the optimizer to escape sharp local minima and explore flatter regions of the loss landscape that tend to generalize better.

The warm restart also provides an elegant approach to model selection. At the end of each cosine cycle, the model has converged to a local minimum with a very small learning rate. This is a natural checkpoint: the model at the end of each cycle can be saved and compared. Since each restart nudges the model toward a different minimum, the ensemble of cycle-end checkpoints can sometimes outperform any single checkpoint. This practice, called snapshot ensembling, was introduced alongside SGDR as a way to get multiple trained models for the cost of one training run.

In[10]:

Code

def cosine_with_restarts(t, eta_max, eta_min, T0, T_mult=1):
    T_cur = T0
    t_remaining = t
    while t_remaining >= T_cur:
        t_remaining -= T_cur
        T_cur = int(T_cur * T_mult)
    return eta_min + 0.5 * (eta_max - eta_min) * (
        1 + np.cos(np.pi * t_remaining / T_cur)
    )


steps_restart = np.arange(0, 200)
cosine_fixed = [
    cosine_with_restarts(t, eta_max=0.1, eta_min=1e-4, T0=50, T_mult=1)
    for t in steps_restart
]
cosine_double = [
    cosine_with_restarts(t, eta_max=0.1, eta_min=1e-4, T0=25, T_mult=2)
    for t in steps_restart
]

Out[11]:

Visualization

Line plot of cosine annealing with fixed 50-epoch cycles on log scale over 200 epochs. — Cosine annealing with four 50-epoch cycles over 200 epochs. The schedule returns to the maximum learning rate after epochs 50, 100, and 150. Each restart gives the optimizer a chance to escape the current local minimum and explore the loss landscape.

Line plot of cosine annealing with doubling cycle lengths on log scale over 200 epochs. — Cosine annealing with doubling cycle lengths starting at 25 epochs. Restarts at epochs 25, 75, and 175 begin cycles of 50, 100, and 200 epochs, so the final 25 visible epochs show only the beginning of the longest cycle.

The doubling schedule is useful when you are uncertain about total training time: early cycles provide frequent opportunities to explore, while later cycles devote progressively longer intervals to annealing within a basin. In this 200-epoch window, the 200-epoch cycle has only just begun at epoch 175.