Universal Asymptotic Optimality of Polyak Momentum
We consider the average-case runtime analysis of algorithms for minimizing quadratic objectives. In this setting, and contrary to the more classical worst-case analysis, non-asymptotic convergence rates and optimal algorithms depend on the full spectrum of the Hessian through its expected spectral distribution. Under mild assumptions, we show that these optimal methods converge asymptotically towards Polyak momentum \emph{independently} of the expected spectral density. This makes Polyak momentum universally (i.e., independent of the spectral distribution) asymptotically average-case optimal.
PDF ICML 2020 PDFTasks
Datasets
Add Datasets
introduced or used in this paper
Results from the Paper
Submit
results from this paper
to get state-of-the-art GitHub badges and help the
community compare results to other papers.
Methods
No methods listed for this paper. Add
relevant methods here
