Search Results for author:
Found 20 papers, 1 papers with code
Differentially Private Optimization with Sparse Gradients
Motivated by applications of large embedding models, we study differentially private (DP) optimization problems under sparsity of individual gradients.
Optimization on a Finer Scale: Bounded Local Subgradient Variation Perspective
The resulting class of objective functions encapsulates the classes of objective functions traditionally studied in optimization, which are defined based on either Lipschitz continuity of the objective or H\"{o}lder/Lipschitz continuity of its gradient.
Public-data Assisted Private Stochastic Optimization: Power and Limitations
We also study PA-DP supervised learning with \textit{unlabeled} public samples.
Mirror Descent Algorithms with Nearly Dimension-Independent Rates for Differentially-Private Stochastic Saddle-Point Problems
For convex-concave and first-order-smooth stochastic objectives, our algorithms attain a rate of $\sqrt{\log(d)/n} + (\log(d)^{3/2}/[n\varepsilon])^{1/3}$, where $d$ is the dimension of the problem and $n$ the dataset size.
Differentially Private Non-Convex Optimization under the KL Condition with Optimal Rates
We further show that, without assuming the KL condition, the same gradient descent algorithm can achieve fast convergence to a stationary point when the gradient stays sufficiently large during the run of the algorithm.
An Oblivious Stochastic Composite Optimization Algorithm for Eigenvalue Optimization Problems
For the $L$-smooth case with a feasible set bounded by $D$, we derive a convergence rate of $ O( {L^2 D^2}/{(T^{2}\sqrt{T})} + {(D_0^2+\sigma^2)}/{\sqrt{T}} )$, where $D_0$ is the starting distance to an optimal solution, and $ \sigma^2$ is the stochastic oracle variance.
Differentially Private Algorithms for the Stochastic Saddle Point Problem with Optimal Rates for the Strong Gap
We show that convex-concave Lipschitz stochastic saddle point problems (also known as stochastic minimax optimization) can be solved under the constraint of $(\epsilon,\delta)$-differential privacy with \emph{strong (primal-dual) gap} rate of $\tilde O\big(\frac{1}{\sqrt{n}} + \frac{\sqrt{d}}{n\epsilon}\big)$, where $n$ is the dataset size and $d$ is the dimension of the problem.
Optimal Algorithms for Stochastic Complementary Composite Minimization
Inspired by regularization techniques in statistics and machine learning, we study complementary composite minimization in the stochastic setting.
Faster Rates of Convergence to Stationary Points in Differentially Private Optimization
We provide a new efficient algorithm that finds an $\tilde{O}\big(\big[\frac{\sqrt{d}}{n\varepsilon}\big]^{2/3}\big)$-stationary point in the finite-sum setting, where $n$ is the number of samples.
Differentially Private Generalized Linear Models Revisited
For this case, we close the gap in the existing work and show that the optimal rate is (up to log factors) $\Theta\left(\frac{\Vert w^*\Vert}{\sqrt{n}} + \min\left\{\frac{\Vert w^*\Vert}{\sqrt{n\epsilon}},\frac{\sqrt{\text{rank}}\Vert w^*\Vert}{n\epsilon}\right\}\right)$, where $\text{rank}$ is the rank of the design matrix.
Stochastic Halpern Iteration with Variance Reduction for Stochastic Monotone Inclusions
We study stochastic monotone inclusion problems, which widely appear in machine learning applications, including robust regression and adversarial learning.
Between Stochastic and Adversarial Online Convex Optimization: Improved Regret Bounds via Smoothness
case, our bounds match the rates one would expect from results in stochastic acceleration, and in the fully adversarial case they gracefully deteriorate to match the minimax regret.
Differentially Private Stochastic Optimization: New Results in Convex and Non-Convex Settings
For the $\ell_1$-case with smooth losses and polyhedral constraint, we provide the first nearly dimension independent rate, $\tilde O\big(\frac{\log^{2/3}{d}}{{(n\varepsilon)^{1/3}}}\big)$ in linear time.
Best-Case Lower Bounds in Online Learning
Much of the work in online learning focuses on the study of sublinear upper bounds on the regret.
Optimal Algorithms for Differentially Private Stochastic Monotone Variational Inequalities and Saddle-Point Problems
We show that a stochastic approximation variant of these algorithms attains risk bounds vanishing as a function of the dataset size, with respect to the strong gap function; and a sampling with replacement variant achieves optimal risk bounds with respect to a weak gap function.
The Complexity of Nonconvex-Strongly-Concave Minimax Optimization
In the averaged smooth finite-sum setting, our proposed algorithm improves over previous algorithms by providing a nearly-tight dependence on the condition number.
Non-Euclidean Differentially Private Stochastic Convex Optimization: Optimal Rates in Linear Time
For $2 < p \leq \infty$, we show that existing linear-time constructions for the Euclidean setup attain a nearly optimal excess risk in the low-dimensional regime.
Complementary Composite Minimization, Small Gradients in General Norms, and Applications
We introduce a new algorithmic framework for complementary composite minimization, where the objective function decouples into a (weakly) smooth and a uniformly convex term.
Stability of Stochastic Gradient Descent on Nonsmooth Convex Losses
Our work is the first to address uniform stability of SGD on {\em nonsmooth} convex losses.
Lower Bounds for Parallel and Randomized Convex Optimization
We study the question of whether parallelization in the exploration of the feasible set can be used to speed up convex optimization, in the local oracle model of computation.
