Demon CM

Introduced by Chen et al. in Demon: Improved Neural Network Training with Momentum Decay

Demon CM, or SGD with Momentum and Demon, is the Demon momentum rule applied to SGD with momentum.

$$ \beta_{t} = \beta_{init}\cdot\frac{\left(1-\frac{t}{T}\right)}{\left(1-\beta_{init}\right) + \beta_{init}\left(1-\frac{t}{T}\right)} $$

$$ \theta_{t+1} = \theta_{t} - \eta{g}_{t} + \beta_{t}v_{t} $$

$$ v_{t+1} = \beta_{t}{v_{t}} - \eta{g_{t}} $$

Source: Demon: Improved Neural Network Training with Momentum Decay

Read Paper See Code

Paper	Code	Results	Date	Stars

Task	Papers	Share
Image Classification	1	100.00%

This feature is experimental; we are continuously improving our matching algorithm.

Component	Type	Add Remove
Demon	Momentum Rules
SGD with Momentum	Stochastic Optimization