Search Results for author:
Found 22 papers, 1 papers with code
Understanding Adam Optimizer via Online Learning of Updates: Adam is FTRL in Disguise
In this work, we provide a different perspective based on online learning that underscores the importance of Adam's algorithmic components.
Linear attention is (maybe) all you need (to understand transformer optimization)
Transformer training is notoriously difficult, requiring a careful design of optimizers and use of various heuristics.
A Unified Approach to Controlling Implicit Regularization via Mirror Descent
However, the implicit regularization of different algorithms are confined to either a specific geometry or a particular class of learning problems, indicating a gap in a general approach for controlling the implicit regularization.
How to escape sharp minima with random perturbations
Under this notion, we then analyze algorithms that find approximate flat minima efficiently.
Learning threshold neurons via the "edge of stability"
For these models, we provably establish the edge of stability phenomenon and discover a sharp phase transition for the step size below which the neural network fails to learn "threshold-like" neurons (i. e., neurons with a non-zero first-layer bias).
Model Predictive Control via On-Policy Imitation Learning
Recent approaches to data-driven MPC have used the simplest form of imitation learning known as behavior cloning to learn controllers that mimic the performance of MPC by online sampling of the trajectories of the closed-loop MPC system.
One-Pass Learning via Bridging Orthogonal Gradient Descent and Recursive Least-Squares
While deep neural networks are capable of achieving state-of-the-art performance in various domains, their training typically requires iterating for many passes over the dataset.
Mirror Descent Maximizes Generalized Margin and Can Be Implemented Efficiently
Driven by the empirical success and wide use of deep neural networks, understanding the generalization performance of overparameterized models has become an increasingly popular question.
Understanding the unstable convergence of gradient descent
Most existing analyses of (stochastic) gradient descent rely on the condition that for $L$-smooth costs, the step size is less than $2/L$.
Reproducibility in Optimization: Theoretical Framework and Limits
We initiate a formal study of reproducibility in optimization.
Agnostic Learnability of Halfspaces via Logistic Loss
In this paper, we close this gap by constructing a well-behaved distribution such that the global minimizer of the logistic risk over this distribution only achieves $\Omega(\sqrt{\textrm{OPT}})$ misclassification risk, matching the upper bound in (Frei et al., 2021).
Riemannian Perspective on Matrix Factorization
We study the non-convex matrix factorization approach to matrix completion via Riemannian geometry.
Optimal dimension dependence of the Metropolis-Adjusted Langevin Algorithm
Conventional wisdom in the sampling literature, backed by a popular diffusion scaling limit, suggests that the mixing time of the Metropolis-Adjusted Langevin Algorithm (MALA) scales as $O(d^{1/3})$, where $d$ is the dimension.
Efficient constrained sampling via the mirror-Langevin algorithm
We propose a new discretization of the mirror-Langevin diffusion and give a crisp proof of its convergence.
SGD with shuffling: optimal rates without component convexity and large epoch requirements
We study without-replacement SGD for solving finite-sum optimization problems.
Understanding Nesterov's Acceleration via Proximal Point Method
The proximal point method (PPM) is a fundamental method in optimization that is often used as a building block for designing optimization algorithms.
On Tight Convergence Rates of Without-replacement SGD
For solving finite-sum optimization problems, SGD without replacement sampling is empirically shown to outperform SGD.
From Nesterov's Estimate Sequence to Riemannian Acceleration
We control this distortion by developing a novel geometric inequality, which permits us to propose and analyze a Riemannian counterpart to Nesterov's accelerated gradient method.
Binary Rating Estimation with Graph Side Information
Considering a simple correlation model between a rating matrix and a graph, we characterize the sharp threshold on the number of observed entries required to recover the rating matrix (called the optimal sample complexity) as a function of the quality of graph side information (to be detailed).
Hypergraph Spectral Clustering in the Weighted Stochastic Block Model
Our main contribution lies in performance analysis of the poly-time algorithms under a random hypergraph model, which we name the weighted stochastic block model, in which objects and multi-way measures are modeled as nodes and weights of hyperedges, respectively.
Community Recovery in Hypergraphs
The objective of the problem is to cluster data points into distinct communities based on a set of measurements, each of which is associated with the values of a certain number of data points.
