Learning Rate Decay › Comparing Schedules at a Glance
Michael BrenndoerferJanuary 24, 20261 min read
Language AI HandbookAll chapters Part 32 of 67Training Optimization
Learning Rate Decay › Comparing Schedules at a GlanceOne section · Sections in this chapter11 / 15 ↓
Sections in this chapter15
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The table below summarizes the key properties of each schedule to guide selection.
| Schedule | Shape | Best for | Limitations |
|---|---|---|---|
| Step decay | Staircase | CV training, interpretability | Rigid timing, requires epoch tuning |
| Exponential | Smooth monotone | Short runs with a floor | Decays too fast without tuning |
| Inverse sqrt | Fast drop then plateau | Transformer pre-training | Needs warmup configuration |
| Cosine | Smooth S-curve | Fine-tuning, general use | Requires total steps known upfront |
| Cosine + restarts | Cyclic | Exploratory training | Cycle length sensitive |