Search Results for author:
Found 25 papers, 4 papers with code
Sparse Concept Bottleneck Models: Gumbel Tricks in Contrastive Learning
We propose a novel architecture and method of explainable classification with Concept Bottleneck Models (CBMs).
Ranked #1 on
Image Classification
on CUB-200-2011
Activations and Gradients Compression for Model-Parallel Training
We analyze compression methods such as quantization and TopK compression, and also experiment with error compensation techniques.
Optimal Data Splitting in Distributed Optimization for Machine Learning
Therefore, a large amount of research has recently been directed at solving this problem.
Ito Diffusion Approximation of Universal Ito Chains for Sampling, Optimization and Boosting
In this work, we consider rather general and broad class of Markov chains, Ito chains, that look like Euler-Maryama discretization of some Stochastic Differential Equation.
First Order Methods with Markovian Noise: from Acceleration to Variational Inequalities
We present a unified approach for the theoretical analysis of first-order gradient methods for stochastic optimization and variational inequalities.
Sarah Frank-Wolfe: Methods for Constrained Optimization with Best Rates and Practical Features
Moreover, our second approach does not require either large batches or full deterministic gradients, which is a typical weakness of many techniques for finite-sum problems.
Similarity, Compression and Local Steps: Three Pillars of Efficient Communications for Distributed Variational Inequalities
The methods presented in this paper have the best theoretical guarantees of communication complexity and are significantly ahead of other methods for distributed variational inequalities.
SARAH-based Variance-reduced Algorithm for Stochastic Finite-sum Cocoercive Variational Inequalities
In this paper we consider the problem of stochastic finite-sum cocoercive variational inequalities.
Smooth Monotone Stochastic Variational Inequalities and Saddle Point Problems: A Survey
This paper is a survey of methods for solving smooth (strongly) monotone stochastic variational inequalities.
Compression and Data Similarity: Combination of Two Techniques for Communication-Efficient Solving of Distributed Variational Inequalities
Variational inequalities are an important tool, which includes minimization, saddles, games, fixed-point problems.
On Scaled Methods for Saddle Point Problems
Methods with adaptive scaling of different features play a key role in solving saddle point problems, primarily due to Adam's popularity for solving adversarial machine learning problems, including GANS training.
Stochastic Gradient Methods with Preconditioned Updates
There are several algorithms for such problems, but existing methods often work poorly when the problem is badly scaled and/or ill-conditioned, and a primary goal of this work is to introduce methods that alleviate this issue.
Optimal Gradient Sliding and its Application to Distributed Optimization Under Similarity
Finally the method is extended to distributed saddle-problems (under function similarity) by means of solving a class of variational inequalities, achieving lower communication and computation complexity bounds.
Stochastic Gradient Descent-Ascent: Unified Theory and New Efficient Methods
Although variants of the new methods are known for solving minimization problems, they were never considered or analyzed for solving min-max problems and VIPs.
Optimal Algorithms for Decentralized Stochastic Variational Inequalities
Our algorithms are the best among the available literature not only in the decentralized stochastic case, but also in the decentralized deterministic and non-distributed stochastic cases.
Distributed Saddle-Point Problems Under Data Similarity
We study solution methods for (strongly-)convex-(strongly)-concave Saddle-Point Problems (SPPs) over networks of two type--master/workers (thus centralized) architectures and mesh (thus decentralized) networks.
Random-reshuffled SARAH does not need a full gradient computations
The StochAstic Recursive grAdient algoritHm (SARAH) algorithm is a variance reduced variant of the Stochastic Gradient Descent (SGD) algorithm that needs a gradient of the objective function from time to time.
Distributed Methods with Compressed Communication for Solving Variational Inequalities, with Theoretical Guarantees
Due to these considerations, it is important to equip existing methods with strategies that would allow to reduce the volume of transmitted information during training while obtaining a model of comparable quality.
Distributed Saddle-Point Problems Under Similarity
We study solution methods for (strongly-)convex-(strongly)-concave Saddle-Point Problems (SPPs) over networks of two type - master/workers (thus centralized) architectures and meshed (thus decentralized) networks.
Decentralized Local Stochastic Extra-Gradient for Variational Inequalities
We extend the stochastic extragradient method to this very general setting and theoretically analyze its convergence rate in the strongly-monotone, monotone, and non-monotone (when a Minty solution exists) settings.
Decentralized Personalized Federated Learning for Min-Max Problems
Personalized Federated Learning (PFL) has witnessed remarkable advancements, enabling the development of innovative machine learning applications that preserve the privacy of training data.
Distributed Saddle-Point Problems: Lower Bounds, Near-Optimal and Robust Algorithms
This paper focuses on the distributed optimization of stochastic saddle point problems.
Zeroth-Order Algorithms for Smooth Saddle-Point Problems
In particular, our analysis shows that in the case when the feasible set is a direct product of two simplices, our convergence rate for the stochastic term is only by a $\log n$ factor worse than for the first-order methods.
Gradient-Free Methods for Saddle-Point Problem
In the second part of the paper, we analyze the case when such an assumption cannot be made, we propose a general approach on how to modernize the method to solve this problem, and also we apply this approach to particular cases of some classical sets.
On Biased Compression for Distributed Learning
In the last few years, various communication compression techniques have emerged as an indispensable tool helping to alleviate the communication bottleneck in distributed learning.
