Learning Rate Schedules

Learning Rate Schedules refer to schedules for the learning rate during the training of neural networks. Below you can find a continuously updating list of learning rate schedules.

METHOD YEAR PAPERS
Linear Warmup With Linear Decay
2000 1152
Linear Warmup With Cosine Annealing
2000 161
Step Decay
2000 67
Exponential Decay
2000 34
Cosine Annealing
2016 32
Slanted Triangular Learning Rates
2018 19
Polynomial Rate Decay
2000 9
Linear Warmup
2000 7
Inverse Square Root Schedule
2000 7
Cyclical Learning Rate Policy
2000 3
1cycle
2018 2