Search Results for author:
Found 61 papers, 18 papers with code
Sparse Concept Bottleneck Models: Gumbel Tricks in Contrastive Learning
We propose a novel architecture and method of explainable classification with Concept Bottleneck Models (CBMs).
Ranked #1 on
Image Classification
on CUB-200-2011
Optimal Flow Matching: Learning Straight Trajectories in Just One Step
Over the several recent years, there has been a boom in development of flow matching methods for generative modeling.
AdaBatchGrad: Combining Adaptive Batch Size and Adaptive Step Size
To substantiate the efficacy of our method, we experimentally show, how the introduction of adaptive step size and adaptive batch size gradually improves the performance of regular SGD.
Optimal Data Splitting in Distributed Optimization for Machine Learning
Therefore, a large amount of research has recently been directed at solving this problem.
Activations and Gradients Compression for Model-Parallel Training
We analyze compression methods such as quantization and TopK compression, and also experiment with error compensation techniques.
Breaking the Heavy-Tailed Noise Barrier in Stochastic Optimization Problems
We consider stochastic optimization problems with heavy-tailed noise with structured density.
High-Probability Convergence for Composite and Distributed Stochastic Minimization and Variational Inequalities with Heavy-Tailed Noise
High-probability analysis of stochastic first-order optimization methods under mild assumptions on the noise has been gaining a lot of attention in recent years.
First Order Methods with Markovian Noise: from Acceleration to Variational Inequalities
We present a unified approach for the theoretical analysis of first-order gradient methods for stochastic optimization and variational inequalities.
Implicitly normalized forecaster with clipping for linear and non-linear heavy-tailed multi-armed bandits
We establish convergence results under mild assumptions on the rewards distribution and demonstrate that INF-clip is optimal for linear heavy-tailed stochastic MAB problems and works well for non-linear ones.
Similarity, Compression and Local Steps: Three Pillars of Efficient Communications for Distributed Variational Inequalities
The methods presented in this paper have the best theoretical guarantees of communication complexity and are significantly ahead of other methods for distributed variational inequalities.
High-Probability Bounds for Stochastic Optimization and Variational Inequalities: the Case of Unbounded Variance
During recent years the interest of optimization and machine learning communities in high-probability convergence of stochastic optimization methods has been growing.
An Optimal Algorithm for Strongly Convex Min-min Optimization
In this paper we study the smooth strongly convex minimization problem $\min_{x}\min_y f(x, y)$.
SARAH-based Variance-reduced Algorithm for Stochastic Finite-sum Cocoercive Variational Inequalities
In this paper we consider the problem of stochastic finite-sum cocoercive variational inequalities.
Smooth Monotone Stochastic Variational Inequalities and Saddle Point Problems: A Survey
This paper is a survey of methods for solving smooth (strongly) monotone stochastic variational inequalities.
Compression and Data Similarity: Combination of Two Techniques for Communication-Efficient Solving of Distributed Variational Inequalities
Variational inequalities are an important tool, which includes minimization, saddles, games, fixed-point problems.
On Scaled Methods for Saddle Point Problems
Methods with adaptive scaling of different features play a key role in solving saddle point problems, primarily due to Adam's popularity for solving adversarial machine learning problems, including GANS training.
Algorithm for Constrained Markov Decision Process with Linear Convergence
The problem of constrained Markov decision process is considered.
Clipped Stochastic Methods for Variational Inequalities with Heavy-Tailed Noise
In this work, we prove the first high-probability complexity results with logarithmic dependence on the confidence level for stochastic methods for solving monotone and structured non-monotone VIPs with non-sub-Gaussian (heavy-tailed) noise and unbounded domains.
Optimal Gradient Sliding and its Application to Distributed Optimization Under Similarity
Finally the method is extended to distributed saddle-problems (under function similarity) by means of solving a class of variational inequalities, achieving lower communication and computation complexity bounds.
The First Optimal Acceleration of High-Order Methods in Smooth Convex Optimization
Arjevani et al. (2019) established the lower bound $\Omega\left(\epsilon^{-2/(3p+1)}\right)$ on the number of the $p$-th order oracle calls required by an algorithm to find an $\epsilon$-accurate solution to the problem, where the $p$-th order oracle stands for the computation of the objective function value and the derivatives up to the order $p$.
The First Optimal Algorithm for Smooth and Strongly-Convex-Strongly-Concave Minimax Optimization
However, the existing state-of-the-art methods do not match this lower bound: algorithms of Lin et al. (2020) and Wang and Li (2020) have gradient evaluation complexity $\mathcal{O}\left( \sqrt{\kappa_x\kappa_y}\log^3\frac{1}{\epsilon}\right)$ and $\mathcal{O}\left( \sqrt{\kappa_x\kappa_y}\log^3 (\kappa_x\kappa_y)\log\frac{1}{\epsilon}\right)$, respectively.
Optimal Algorithms for Decentralized Stochastic Variational Inequalities
Our algorithms are the best among the available literature not only in the decentralized stochastic case, but also in the decentralized deterministic and non-distributed stochastic cases.
Accelerated Primal-Dual Gradient Method for Smooth and Convex-Concave Saddle-Point Problems with Bilinear Coupling
In this paper we study the convex-concave saddle-point problem $\min_x \max_y f(x) + y^T \mathbf{A} x - g(y)$, where $f(x)$ and $g(y)$ are smooth and convex functions.
Lower Bounds and Optimal Algorithms for Smooth and Strongly Convex Decentralized Optimization Over Time-Varying Networks
We consider the task of minimizing the sum of smooth and strongly convex functions stored in a decentralized manner across the nodes of a communication network whose links are allowed to change in time.
Distributed Saddle-Point Problems Under Data Similarity
We study solution methods for (strongly-)convex-(strongly)-concave Saddle-Point Problems (SPPs) over networks of two type--master/workers (thus centralized) architectures and mesh (thus decentralized) networks.
Acceleration in Distributed Optimization under Similarity
In order to reduce the number of communications to reach a solution accuracy, we proposed a {\it preconditioned, accelerated} distributed method.
Distributed Methods with Compressed Communication for Solving Variational Inequalities, with Theoretical Guarantees
Due to these considerations, it is important to equip existing methods with strategies that would allow to reduce the volume of transmitted information during training while obtaining a model of comparable quality.
Distributed Saddle-Point Problems Under Similarity
We study solution methods for (strongly-)convex-(strongly)-concave Saddle-Point Problems (SPPs) over networks of two type - master/workers (thus centralized) architectures and meshed (thus decentralized) networks.
Decentralized Local Stochastic Extra-Gradient for Variational Inequalities
We extend the stochastic extragradient method to this very general setting and theoretically analyze its convergence rate in the strongly-monotone, monotone, and non-monotone (when a Minty solution exists) settings.
Decentralized Personalized Federated Learning for Min-Max Problems
Personalized Federated Learning (PFL) has witnessed remarkable advancements, enabling the development of innovative machine learning applications that preserve the privacy of training data.
Near-Optimal High Probability Complexity Bounds for Non-Smooth Stochastic Optimization with Heavy-Tailed Noise
In our paper, we resolve this issue and derive the first high-probability convergence results with logarithmic dependence on the confidence level for non-smooth convex stochastic optimization problems with non-sub-Gaussian (heavy-tailed) noise.
Gradient Clipping Helps in Non-Smooth Stochastic Optimization with Heavy-Tailed Noise
In our paper, we resolve this issue and derive the first high-probability convergence results with logarithmical dependence on the confidence level for non-smooth convex stochastic optimization problems with non-sub-Gaussian (heavy-tailed) noise.
Primal-Dual Stochastic Mirror Descent for MDPs
We consider the problem of learning the optimal policy for infinite-horizon Markov decision processes (MDPs).
ADOM: Accelerated Decentralized Optimization Method for Time-Varying Networks
We propose ADOM - an accelerated method for smooth and strongly convex decentralized optimization over time-varying networks.
Hyperfast Second-Order Local Solvers for Efficient Statistically Preconditioned Distributed Optimization
Statistical preconditioning enables fast methods for distributed large-scale empirical risk minimization problems.
Distributed Optimization
Optimization and Control
Decentralized Distributed Optimization for Saddle Point Problems
We consider distributed convex-concave saddle point problems over arbitrary connected undirected networks and propose a decentralized distributed algorithm for their solution.
Distributed Optimization
Optimization and Control
Distributed, Parallel, and Cluster Computing
Flexible Modification of Gauss-Newton Method and Its Stochastic Extension
This work presents a novel version of recently developed Gauss-Newton method for solving systems of nonlinear equations, based on upper bound of solution residual and quadratic regularization ideas.
Stochastic Optimization
Optimization and Control
Improved Exploiting Higher Order Smoothness in Derivative-free Optimization and Continuous Bandit
We consider $\beta$-smooth (satisfies the generalized Holder condition with parameter $\beta > 2$) stochastic convex optimization problem with zero-order one-point oracle.
Optimization and Control
Tensor methods for strongly convex strongly concave saddle point problems and strongly monotone variational inequalities
The first method is based on the assumption of $p$-th order smoothness of the objective and it achieves a convergence rate of $O \left( \left( \frac{L_p R^{p - 1}}{\mu} \right)^\frac{2}{p + 1} \log \frac{\mu R^2}{\varepsilon_G} \right)$, where $R$ is an estimate of the initial distance to the solution, and $\varepsilon_G$ is the error in terms of duality gap.
Optimization and Control
Inexact Tensor Methods and Their Application to Stochastic Convex Optimization
We propose general non-accelerated and accelerated tensor methods under inexact information on the derivatives of the objective, analyze their convergence rate.
Optimization and Control
Recent Theoretical Advances in Non-Convex Optimization
For this setting, we first present known results for the convergence rates of deterministic first-order methods, which are then followed by a general theoretical analysis of optimal stochastic and randomized gradient schemes, and an overview of the stochastic first-order methods.
Finding equilibrium in two-stage traffic assignment model
The first block consists of model for calculating correspondence (demand) matrix, whereas the second block is a traffic assignment model.
Optimization and Control
Distributed Saddle-Point Problems: Lower Bounds, Near-Optimal and Robust Algorithms
This paper focuses on the distributed optimization of stochastic saddle point problems.
Zeroth-Order Algorithms for Smooth Saddle-Point Problems
In particular, our analysis shows that in the case when the feasible set is a direct product of two simplices, our convergence rate for the stochastic term is only by a $\log n$ factor worse than for the first-order methods.
Finding Equilibria in the Traffic Assignment Problem with Primal-Dual Gradient Methods for Stable Dynamics Model and Beckmann Model
In this paper we consider the application of several gradient methods to the traffic assignment problem: we search equilibria in the stable dynamics model (Nesterov and De Palma, 2003) and the Beckmann model.
Optimization and Control
Stochastic Saddle-Point Optimization for Wasserstein Barycenters
This leads to a complicated stochastic optimization problem where the objective is given as an expectation of a function given as a solution to a random optimization problem.
Stochastic Optimization with Heavy-Tailed Noise via Accelerated Gradient Clipping
In this paper, we propose a new accelerated stochastic first-order method called clipped-SSTM for smooth convex stochastic optimization with heavy-tailed distributed noise in stochastic gradients and derive the first high-probability complexity bounds for this method closing the gap in the theory of stochastic optimization with heavy-tailed noise.
Gradient-Free Methods for Saddle-Point Problem
In the second part of the paper, we analyze the case when such an assumption cannot be made, we propose a general approach on how to modernize the method to solve this problem, and also we apply this approach to particular cases of some classical sets.
Accelerated meta-algorithm for convex optimization
We propose an accelerated meta-algorithm, which allows to obtain accelerated methods for convex unconstrained minimization in different settings.
Optimization and Control
Adaptive Catalyst for Smooth Convex Optimization
In this paper, we present a generic framework that allows accelerating almost arbitrary non-accelerated deterministic and randomized algorithms for smooth convex optimization problems.
Optimization and Control
Adaptive Gradient Descent for Convex and Non-Convex Stochastic Optimization
In this paper we propose several adaptive gradient methods for stochastic optimization.
Optimization and Control
On a Combination of Alternating Minimization and Nesterov's Momentum
In this paper we combine AM and Nesterov's acceleration to propose an accelerated alternating minimization algorithm.
A Dual Approach for Optimal Algorithms in Distributed Optimization over Networks
Then, we study distributed optimization algorithms for non-dual friendly functions, as well as a method to improve the dependency on the parameters of the functions involved.
Walrasian Equilibrium and Centralized Distributed Optimization from the point of view of Modern Convex Optimization Methods on the Example of Resource Allocation Problem
We consider the resource allocation problem and its numerical solution.
Optimization and Control 90B99
On the upper bound for the mathematical expectation of the norm of a vector uniformly distributed on the sphere and the phenomenon of concentration of uniform measure on the sphere
We considered the problem of obtaining upper bounds for the mathematical expectation of the $q$-norm ($2\leqslant q \leqslant \infty$) of the vector which is uniformly distributed on the unit Euclidean sphere.
Optimization and Control
Distributed Computation of Wasserstein Barycenters over Networks
We propose a new \cu{class-optimal} algorithm for the distributed computation of Wasserstein Barycenters over networks.
An Accelerated Method for Derivative-Free Smooth Stochastic Convex Optimization
In the two-point feedback setting, i. e. when pairs of function values are available, we propose an accelerated derivative-free algorithm together with its complexity analysis.
Optimization and Control Computational Complexity
Computational Optimal Transport: Complexity by Accelerated Gradient Descent Is Better Than by Sinkhorn's Algorithm
We analyze two algorithms for approximating the general optimal transport (OT) distance between two discrete distributions of size $n$, up to accuracy $\varepsilon$.
Data Structures and Algorithms Optimization and Control
Optimal Algorithms for Distributed Optimization
In this paper, we study the optimal convergence rate for distributed convex optimization problems in networks.
Accelerated Directional Search with non-Euclidean prox-structure
In the paper we show how to make Nesterov's method $n$-times faster (up to a $\log n$-factor) in this case.
Optimization and Control
Learning Supervised PageRank with Gradient-Based and Gradient-Free Optimization Methods
In this paper, we consider a non-convex loss-minimization problem of learning Supervised PageRank models, which can account for features of nodes and edges.
