SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum

1 code implementation • ICLR 2020 • Jianyu Wang, Vinayak Tantia, Nicolas Ballas, Michael Rabbat

We provide theoretical convergence guarantees showing that SlowMo converges to a stationary point of smooth non-convex losses.

Blocking Distributed Optimization +3

A Modern Take on the Bias-Variance Tradeoff in Neural Networks

no code implementations • 19 Oct 2018 • Brady Neal, Sarthak Mittal, Aristide Baratin, Vinayak Tantia, Matthew Scicluna, Simon Lacoste-Julien, Ioannis Mitliagkas

The bias-variance tradeoff tells us that as model complexity increases, bias falls and variances increases, leading to a U-shaped test error curve.