Learning Rate Decay › Choosing a Schedule

No single schedule is universally best. The right choice depends on the model architecture, optimizer, dataset size, and training budget. What follows is a practical guide to making this choice, organized by use case.

For training large transformer models from scratch, the inverse square root schedule with linear warmup is the canonical default. It was validated on the original Transformer and remains widely used for pre-training. The slow later decay means the model continues to see meaningful learning rate values even late in training.

The warmup length for large models is typically 4,000 to 10,000 steps. The peak learning rate is usually in the range of $10^{-4}$ to $5 \times 10^{-4}$ for model dimensions around 512-1024. For larger models, the peak is often scaled inversely with the square root of the model dimension, following the Transformer paper's formula.

Some recent large language model training runs have moved away from the inverse square root schedule in favor of cosine annealing with a fixed total step count. The argument is that cosine annealing with a known endpoint is more predictable and allows planning the training budget in advance. The inverse square root schedule is "endless" in the sense that there is no natural stopping point, which complicates budget planning.

For fine-tuning pre-trained models, linear or cosine decay is preferred because fine-tuning involves much shorter training runs. The learning rate needs to reach a very small value by the end of training to avoid disrupting the pre-trained weights. Cosine decay provides a smooth, predictable endpoint.

The peak learning rate for fine-tuning is typically much smaller than for pre-training. For BERT-style models, common values are $1 \times 10^{-5}$ to $5 \times 10^{-5}$. For larger models like GPT-style architectures, the fine-tuning learning rate is often even smaller, sometimes as low as $10^{-6}$, to prevent catastrophic forgetting of pre-trained knowledge.

The warmup period for fine-tuning is shorter than for pre-training, typically 5 to 10 percent of total steps. Since the model's weights are already in a reasonable part of parameter space (thanks to pre-training), the risk of early instability is lower. The warmup mainly serves to let Adam's moment estimates calibrate.

For computer vision with SGD, step decay remains common. The explicit phase structure matches the training recipes for ResNet and related architectures, and the abrupt drops are known to produce good convergence behavior.

The standard three-phase recipe for ImageNet training is: 90 epochs total, starting learning rate of 0.1, drops of 10x at epochs 30 and 60. This recipe has been so successful and widely validated that it has become a community reference point. Many papers compare their training protocols against this baseline.

When you switch from SGD to AdamW for computer vision, the step decay recipe often needs adjustment. Adam's internal adaptation changes the effective sensitivity to the global schedule, and the ten-fold drops that work well for SGD can be too aggressive for Adam. A cosine schedule with AdamW tends to transfer more reliably across different model scales and training durations.

When in doubt between linear and cosine decay, cosine decay is slightly preferred because it is slower to start falling (reducing the learning rate less aggressively early in decay) and ends more smoothly. In practice, the difference is small compared to the impact of other hyperparameters. If you are setting up a new experiment and have no prior experience with the task, cosine annealing with linear warmup using 5 to 10 percent of total steps is a reasonable starting point.

The most important hyperparameter to tune is the peak learning rate, not the schedule shape. A cosine schedule with a poorly tuned peak learning rate will underperform a linear schedule with the right peak learning rate. Spend your tuning budget on the peak learning rate first, and only refine the schedule shape once that is set.