Search Results for author:
Found 30 papers, 5 papers with code
Complexity of Finding Stationary Points of Nonconvex Nonsmooth Functions
Therefore, we introduce the notion of (delta, epsilon)-stationarity, a generalization that allows for a point to be within distance delta of an epsilon-stationary point and reduces to epsilon-stationarity for smooth functions.
Efficient Sampling on Riemannian Manifolds via Langevin MCMC
We study the task of efficiently sampling from a Gibbs distribution $d \pi^* = e^{-h} d {vol}_g$ over a Riemannian manifold $M$ via (geometric) Langevin MCMC; this algorithm involves computing exponential maps in random Gaussian directions and is efficiently implementable in practice.
A Quadratic Synchronization Rule for Distributed Deep Learning
In distributed deep learning with data parallelism, synchronizing gradients at each training step can cause a huge communication overhead, especially when many nodes work together to train large models.
Two Phases of Scaling Laws for Nearest Neighbor Classifiers
We show that a scaling law can have two phases: in the first phase, the generalization error depends polynomially on the data dimension and decreases fast; whereas in the second phase, the error depends exponentially on the data dimension and decreases slowly.
Near-Optimal Nonconvex-Strongly-Convex Bilevel Optimization with Fully First-Order Oracles
Designing efficient algorithms for bilevel optimization is challenging because the lower-level problem defines a feasibility set implicitly via another optimization problem.
Fast Conditional Mixing of MCMC Algorithms for Non-log-concave Distributions
However, on the theory side, MCMC algorithms suffer from slow mixing rate when $\pi(x)$ is non-log-concave.
Lower Generalization Bounds for GD and SGD in Smooth Stochastic Convex Optimization
Our paper examines this observation by providing excess risk lower bounds for GD and SGD in two realizable settings: 1) $\eta T = \bigO{n}$, and (2) $\eta T = \bigOmega{n}$, where $n$ is the size of dataset.
On Finding Small Hyper-Gradients in Bilevel Optimization: Hardness Results and Improved Analysis
Bilevel optimization reveals the inner structure of otherwise oblique optimization problems, such as hyperparameter tuning, neural architecture search, and meta-learning.
Online Policy Optimization for Robust MDP
Reinforcement learning (RL) has exceeded human performance in many synthetic settings such as video games and Go.
Benign Overfitting in Classification: Provably Counter Label Noise with Larger Models
Studies on benign overfitting provide insights for the success of overparameterized deep learning models.
Understanding the unstable convergence of gradient descent
Most existing analyses of (stochastic) gradient descent rely on the condition that for $L$-smooth costs, the step size is less than $2/L$.
Sion's Minimax Theorem in Geodesic Metric Spaces and a Riemannian Extragradient Algorithm
Deciding whether saddle points exist or are approximable for nonconvex-nonconcave problems is usually intractable.
EVBattery: A Large-Scale Electric Vehicle Dataset for Battery Health and Capacity Estimation
In addition to demonstrating how existing deep learning algorithms can be applied to this task, we further develop an algorithm that exploits the data structure of battery systems.
Neural Network Weights Do Not Converge to Stationary Points: An Invariant Measure Perspective
This work examines the deep disconnect between existing theoretical analyses of gradient-based algorithms and the practice of training deep neural networks.
Fast Federated Learning in the Presence of Arbitrary Device Unavailability
In this case, the convergence of popular FL algorithms such as FedAvg is severely influenced by the straggling devices.
Complexity Lower Bounds for Nonconvex-Strongly-Concave Min-Max Optimization
We provide a first-order oracle complexity lower bound for finding stationary points of min-max optimization problems where the objective function is smooth, nonconvex in the minimization variable, and strongly concave in the maximization variable.
Provably Efficient Algorithms for Multi-Objective Competitive RL
To our knowledge, this work provides the first provably efficient algorithms for vector-valued Markov games and our theoretical guarantees are near-optimal.
Stochastic Optimization with Non-stationary Noise: The Power of Moment Estimation
In particular, standard results on optimal convergence rates for stochastic optimization assume either there exists a uniform bound on the moments of the gradient noise, or that the noise decays as the algorithm progresses.
On the Overlooked Pitfalls of Weight Decay and How to Mitigate Them: A Gradient-Norm Perspective
Weight decay is a simple yet powerful regularization technique that has been very widely used in training of deep neural networks (DNNs).
Coping with Label Shift via Distributionally Robust Optimisation
The label shift problem refers to the supervised learning setting where the train and test label distributions do not match.
Quantifying Exposure Bias for Open-ended Language Generation
The exposure bias problem refers to the incrementally distorted generation induced by the training-generation discrepancy, in teacher-forcing training for auto-regressive neural network language models (LM).
Beyond Worst-Case Analysis in Stochastic Approximation: Moment Estimation Improves Instance Complexity
We study oracle complexity of gradient based methods for stochastic approximation problems.
Complexity of Finding Stationary Points of Nonsmooth Nonconvex Functions
In particular, we study the class of Hadamard semi-differentiable functions, perhaps the largest class of nonsmooth functions for which the chain rule of calculus holds.
Why are Adaptive Methods Good for Attention Models?
While stochastic gradient descent (SGD) is still the \emph{de facto} algorithm in deep learning, adaptive methods like Clipped SGD/Adam have been observed to outperform SGD across important tasks, such as attention models.
Why ADAM Beats SGD for Attention Models
While stochastic gradient descent (SGD) is still the de facto algorithm in deep learning, adaptive methods like Adam have been observed to outperform SGD across important tasks, such as attention models.
Why gradient clipping accelerates training: A theoretical justification for adaptivity
We provide a theoretical explanation for the effectiveness of gradient clipping in training deep neural networks.
Exposure Bias versus Self-Recovery: Are Distortions Really Incremental for Autoregressive Text Generation?
Exposure bias has been regarded as a central problem for auto-regressive language models (LM).
A Probe Towards Understanding GAN and VAE Models
This project report compares some known GAN and VAE models proposed prior to 2017.
R-SPIDER: A Fast Riemannian Stochastic Optimization Algorithm with Curvature Independent Rate
We study smooth stochastic optimization problems on Riemannian manifolds.
Direct Runge-Kutta Discretization Achieves Acceleration
We study gradient-based optimization methods obtained by directly discretizing a second-order ordinary differential equation (ODE) related to the continuous limit of Nesterov's accelerated gradient method.
