Search Results for author:
Found 5 papers, 1 papers with code
Scaling the Wild: Decentralizing Hogwild!-style Shared-memory SGD
Our scheme is based on the following algorithmic tools and features: (a) asynchronous local gradient updates on the shared-memory of workers, (b) partial backpropagation, and (c) non-blocking in-place averaging of the local models.
Local SGD Meets Asynchrony
On the theoretical side, we show that this method guarantees ergodic convergence for non-convex objectives, and achieves the classic sublinear rate under standard assumptions.
Stochastic Gradient Langevin with Delayed Gradients
Stochastic Gradient Langevin Dynamics (SGLD) ensures strong guarantees with regards to convergence in measure for sampling log-concave posterior distributions by adding noise to stochastic gradient iterates.
Elastic Consistency: A General Consistency Model for Distributed Stochastic Gradient Descent
Our framework, called elastic consistency enables us to derive convergence bounds for a variety of distributed SGD methods used in practice to train large-scale machine learning models.
Asynchronous Stochastic Subgradient Methods for General Nonsmooth Nonconvex Optimization
This is all the more surprising since these objectives are the ones appearing in the training of deep neural networks.
