Search Results for author:
Found 19 papers, 4 papers with code
Layer-wise Adaptive Step-Sizes for Stochastic First-Order Methods for Deep Learning
We propose a new per-layer adaptive step-size procedure for stochastic first-order optimization methods for minimizing empirical loss functions in deep learning, eliminating the need for the user to tune the learning rate (LR).
A Mini-Block Fisher Method for Deep Neural Networks
Specifically, our method uses a block-diagonal approximation to the empirical Fisher matrix, where for each layer in the DNN, whether it is convolutional or feed-forward and fully connected, the associated diagonal block is itself block-diagonal and is composed of a large number of mini-blocks of modest size.
Tensor Normal Training for Deep Learning Models
Based on the so-called tensor normal (TN) distribution, we propose and analyze a brand new approximate natural gradient method, Tensor Normal Training (TNT), which like Shampoo, only requires knowledge of the shape of the training parameters.
Kronecker-factored Quasi-Newton Methods for Deep Learning
Several improvements to the methods in Goldfarb et al. (2020) are also proposed that can be applied to both MLPs and CNNs.
Practical Quasi-Newton Methods for Training Deep Neural Networks
We consider the development of practical stochastic quasi-Newton, and in particular Kronecker-factored block-diagonal BFGS and L-BFGS methods, for training deep neural networks (DNNs).
A Dynamic Sampling Adaptive-SGD Method for Machine Learning
We also propose an adaptive version of ADAM that eliminates the need to tune the base learning rate and compares favorably to fine-tuned ADAM on training DNNs.
Efficient Subsampled Gauss-Newton and Natural Gradient Methods for Training Neural Networks
We present practical Levenberg-Marquardt variants of Gauss-Newton and natural gradient methods for solving non-convex optimization problems that arise in training deep neural networks involving enormous numbers of variables and huge data sets.
Leader Stochastic Gradient Descent for Distributed Training of Deep Learning Models: Extension
Finally, we implement an asynchronous version of our algorithm and extend it to the multi-leader setting, where we form groups of workers, each represented by its own local leader (the best performer in a group), and update each worker with a corrective direction comprised of two attractive forces: one to the local, and one to the global leader (the best performer among all workers).
Increasing Iterate Averaging for Solving Saddle-Point Problems
In particular, the increasing averages consistently outperform the uniform averages in all test problems by orders of magnitude.
Stochastic Adaptive Quasi-Newton Methods for Minimizing Expected Values
We propose a novel class of stochastic, adaptive methods for minimizing self-concordant functions which can be expressed as an expected value.
Stochastic Quasi-Newton Methods for Nonconvex Stochastic Optimization
In this paper we study stochastic quasi-Newton methods for nonconvex stochastic optimization, where we assume that noisy information about the gradients of the objective function is available via a stochastic first-order oracle (SFO).
Scalable Robust Matrix Recovery: Frank-Wolfe Meets Proximal Methods
Recovering matrices from compressive and grossly corrupted observations is a fundamental problem in robust statistics, with rich applications in computer vision and machine learning.
Robust Low-rank Tensor Recovery: Models and Algorithms
Robust tensor recovery plays an instrumental role in robustifying tensor decompositions for multilinear data analysis against outliers, gross corruptions and missing values and has a diverse array of applications.
Efficient Algorithms for Robust and Stable Principal Component Pursuit Problems
Moreover, if the observed data matrix has also been corrupted by a dense noise matrix in addition to gross sparse error, then the stable principal component pursuit (SPCP) problem is solved to recover the low-rank matrix.
Optimization and Control
Square Deal: Lower Bounds and Improved Relaxations for Tensor Recovery
The most popular convex relaxation of this problem minimizes the sum of the nuclear norms of the unfoldings of the tensor.
Fast First-Order Methods for Stable Principal Component Pursuit
The stable principal component pursuit (SPCP) problem is a non-smooth convex optimization problem, the solution of which has been shown both in theory and in practice to enable one to recover the low rank and sparse components of a matrix whose elements have been corrupted by Gaussian noise.
Optimization and Control
Sparse Inverse Covariance Selection via Alternating Linearization Methods
Gaussian graphical models are of great interest in statistical learning.
Fast Alternating Linearization Methods for Minimizing the Sum of Two Convex Functions
We present in this paper first-order alternating linearization algorithms based on an alternating direction augmented Lagrangian approach for minimizing the sum of two convex functions.
Fixed Point and Bregman Iterative Methods for Matrix Rank Minimization
The tightest convex relaxation of this problem is the linearly constrained nuclear norm minimization.
Optimization and Control Information Theory Information Theory
