Search Results for author:
Found 14 papers, 4 papers with code
Directional Smoothness and Gradient Methods: Convergence and Adaptivity
We develop new sub-optimality bounds for gradient descent (GD) that depend on the conditioning of the objective along the path of optimization, rather than on global, worst-case constants.
Tuning-Free Stochastic Optimization
For the task of finding a stationary point of a smooth and potentially nonconvex function, we give a variant of SGD that matches the best-known high-probability convergence rate for tuned SGD at only an additional polylogarithmic cost.
DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Method
This paper proposes a new easy-to-implement parameter-free gradient-based optimizer: DoWG (Distance over Weighted Gradients).
Faster federated optimization under second-order similarity
Federated learning (FL) is a subfield of machine learning where multiple clients try to collaboratively learn a model over a network under communication constraints.
Federated Optimization Algorithms with Random Reshuffling and Gradient Compression
To reveal the true advantages of RR in the distributed learning with compression, we propose a new method called DIANA-RR that reduces the compression variance and has provably better convergence rates than existing counterparts with with-replacement sampling of stochastic gradients.
FLIX: A Simple and Communication-Efficient Alternative to Local Methods in Federated Learning
A persistent problem in federated learning is that it is not clear what the optimization objective should be: the standard average risk minimization of supervised learning is inadequate in handling several major constraints specific to federated learning, such as communication adaptivity and personalization control.
Proximal and Federated Random Reshuffling
Random Reshuffling (RR), also known as Stochastic Gradient Descent (SGD) without replacement, is a popular and theoretically grounded method for finite-sum minimization.
Unified Analysis of Stochastic Gradient Methods for Composite Convex and Smooth Optimization
We showcase this by obtaining a simple formula for the optimal minibatch size of two variance reduced methods (\textit{L-SVRG} and \textit{SAGA}).
Random Reshuffling: Simple Analysis with Vast Improvements
from $\kappa$ to $\sqrt{\kappa}$) and, in addition, show that RR has a different type of variance.
Better Theory for SGD in the Nonconvex World
Moreover, we perform our analysis in a framework which allows for a detailed study of the effects of a wide array of sampling strategies and minibatch sizes for finite-sum optimization problems.
Distributed Fixed Point Methods with Compressed Iterates
We propose basic and natural assumptions under which iterative optimization methods with compressed iterates can be analyzed.
Gradient Descent with Compressed Iterates
We propose and analyze a new type of stochastic first order method: gradient descent with compressed iterates (GDCI).
First Analysis of Local GD on Heterogeneous Data
We provide the first convergence analysis of local gradient descent for minimizing the average of smooth and convex but otherwise arbitrary functions.
Tighter Theory for Local SGD on Identical and Heterogeneous Data
We provide a new analysis of local SGD, removing unnecessary assumptions and elaborating on the difference between two data regimes: identical and heterogeneous.
