Learning Rate Schedules

Learning Rate Schedules refer to schedules for the learning rate during the training of neural networks. Below you can find a continuously updating list of learning rate schedules.

METHOD YEAR PAPERS
Linear Warmup With Linear Decay
2000 1413
Linear Warmup With Cosine Annealing
2000 201
Step Decay
2000 65
Cosine Annealing
2016 34
Exponential Decay
2000 34
Slanted Triangular Learning Rates
2018 23
Inverse Square Root Schedule
2000 16
Polynomial Rate Decay
2000 9
Linear Warmup
2000 6
Cyclical Learning Rate Policy
2000 3
1cycle
2018 2