Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Cosine Learning Rate Schedule › Cosine vs. Linear Decay

Michael BrenndoerferJanuary 26, 20265 min read

Language AI HandbookAll chapters

Part 32 of 67Training Optimization

Chapter 233Cosine Learning Rate Schedule › Cosine vs. Linear Decay

Reading section 6 of 14 · Cosine vs. Linear Decay · 5 minSection 6 of 14 · 5 min

Section 6 of 14 · 5 min read

Cosine Learning Rate Schedule › Cosine vs. Linear DecayOne section · Sections in this chapter6 / 14 ↓

Sections in this chapter14

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The choice between cosine and linear decay often appears in ablation studies and reproducibility discussions. Both decay from $\eta_{\max}$ to $\eta_{\min}$ , but they allocate the budget differently across training steps.

Linear decay decreases the learning rate by the same absolute amount at every step. Starting from $\eta_{\max}$ at $t = 0$ and ending at $\eta_{\min}$ at $t = T$ , the linear schedule computes:

\eta_{\text{linear}}(t) = \eta_{\max} - \frac{(\eta_{\max} - \eta_{\min}) \cdot t}{T}

where:

$\eta_{\max} - \eta_{\min}$ : the total decrease in learning rate over training
$t / T$ : the fraction of training completed, ranging from 0 to 1

At any step $t$ , the learning rate has decreased by exactly the fraction $t/T$ of the total budget.

Cosine decay is concave (curves downward) in the first half of training and convex (curves upward) in the second half. The difference between cosine and linear at any step $t$ is:

\Delta(t) = \eta_{\text{cosine}}(t) - \eta_{\text{linear}}(t) = \frac{1}{2}(\eta_{\max} - \eta_{\min})\left(\cos\left(\frac{\pi t}{T}\right) - 1 + \frac{2t}{T}\right)

This difference $\Delta(t)$ is positive in the first half of training (cosine is above linear) and negative in the second half (cosine is below linear), with $\Delta(0) = \Delta(T) = 0$ . The cosine schedule thus "front-loads" exploration by keeping the rate higher early on, then "back-loads" convergence by bringing the rate below the linear trajectory in the final phase.

The practical consequence is that cosine training often reaches a slightly lower final loss on the same number of steps compared to linear decay. The GPT-3, LLaMA, Mistral, and Gemma model families all use cosine schedules in their reported training configurations.

Out[5]:

Visualization

Area chart showing the signed difference between cosine and linear learning rates across 1000 training steps. — Difference between cosine and linear learning rate at each training step, expressed as a percentage of the total decay range. Positive values (first half) indicate that cosine keeps the learning rate higher than linear, favoring exploration. Negative values (second half) show cosine decaying below linear, applying finer updates during convergence. The symmetric shape reflects the cosine function's geometry.

Illustrative Comparison with Simulated Loss CurvesLink Copied

The schedule geometry alone does not guarantee that cosine will outperform linear decay on a real model. A controlled simulation can still illustrate the mechanism: higher rates early reduce optimization bias faster, while lower rates late reduce noise around the minimum. The next example averages 512 paired runs on the same noisy quadratic objective so that both schedules see identical gradient perturbations.

In[6]:

Code

import numpy as np

# Simulate paired SGD runs on a noisy quadratic: L(x) = 0.5 * x^2.
T_sim = 500
num_trials = 512
noise_scale = 0.5
rng_sim = np.random.default_rng(42)
gradient_noise = rng_sim.normal(0, noise_scale, size=(num_trials, T_sim))


def simulate_sgd(schedule_fn, noise_samples, x_init=2.0):
    x = np.full(noise_samples.shape[0], x_init, dtype=float)
    losses = []
    for t in range(noise_samples.shape[1]):
        lr = schedule_fn(t)
        grad = x + noise_samples[:, t]
        x = x - lr * grad
        losses.append(0.5 * np.mean(x**2))
    return np.asarray(losses)


cosine_fn = lambda t: cosine_lr(2.5e-2, 2.5e-3, t, T_sim)
linear_fn = lambda t: linear_lr(2.5e-2, 2.5e-3, t, T_sim)

cosine_losses = simulate_sgd(cosine_fn, gradient_noise)
linear_losses = simulate_sgd(linear_fn, gradient_noise)

Out[7]:

Console

Mean simulated losses across 512 paired noisy-quadratic runs:
  Cosine schedule (avg last 20 steps): 0.000410
  Linear schedule (avg last 20 steps): 0.000459
  Cosine improvement over linear: 10.7%

Out[8]:

Visualization

Log-scale mean loss curves for cosine and linear schedules across 512 paired noisy-quadratic simulations. — Mean loss across 512 paired noisy-quadratic optimization runs over 500 steps, shown on a logarithmic scale. Both schedules receive the same gradient perturbations. Cosine reduces loss faster early because it stays above linear, and its lower late-stage rate produces an average loss about 11% lower over the final 20 steps in this illustrative setup.