Search Results for author:
Found 55 papers, 11 papers with code
Better-than-KL PAC-Bayes Bounds
In this paper, we consider the problem of proving concentration inequalities to estimate the mean of the sequence.
Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosion
We answer this question in the affirmative by giving a particular construction of an Multi-Layer Perceptron (MLP) with linear activations and batch-normalization that provably has bounded gradients at any depth.
Normalized Gradients for All
In this short note, I show how to adapt to H\"{o}lder smoothness using normalized gradients in a black-box way.
Implicit Interpretation of Importance Weight Aware Updates
Due to its speed and simplicity, subgradient descent is one of the most used optimization algorithms in convex machine learning algorithms.
Generalized Implicit Follow-The-Regularized-Leader
We propose a new class of online learning algorithms, generalized implicit Follow-The-Regularized-Leader (FTRL), that expands the scope of FTRL framework.
Tighter PAC-Bayes Bounds Through Coin-Betting
We consider the problem of estimating the mean of a sequence of random elements $f(X_1, \theta)$ $, \ldots, $ $f(X_n, \theta)$ where $f$ is a fixed scalar function, $S=(X_1, \ldots, X_n)$ are independent random variables, and $\theta$ is a possibly $S$-dependent parameter.
Optimal Stochastic Non-smooth Non-convex Optimization through Online-to-Non-convex Conversion
Our primary technique is a reduction from non-smooth non-convex optimization to online learning, after which our results follow from standard regret bounds in online learning.
Robustness to Unbounded Smoothness of Generalized SignSGD
We also compare these algorithms with popular optimizers on a set of deep learning tasks, observing that we can match the performance of Adam while beating the others.
Implicit Parameter-free Online Learning with Truncated Linear Models
Parameter-free algorithms are online learning algorithms that do not require setting learning rates.
Understanding AdamW through Proximal Methods and Scale-Freeness
First, we show how to re-interpret AdamW as an approximation of a proximal gradient method, which takes advantage of the closed-form proximal mapping of the regularizer instead of only utilizing its gradient information as in Adam-$\ell_2$.
Tight Concentrations and Confidence Sequences from the Regret of Universal Portfolio
A classic problem in statistics is the estimation of the expectation of random variables from samples.
Minimax Optimal Quantile and Semi-Adversarial Regret via Root-Logarithmic Regularizers
Quantile (and, more generally, KL) regret bounds, such as those achieved by NormalHedge (Chaudhuri, Freund, and Hsu 2009) and its variants, relax the goal of competing against the best individual expert to only competing against a majority of experts on adversarial data.
Online Learning with Optimism and Delay
Inspired by the demands of real-time climate and weather forecasting, we develop optimistic online learning algorithms that require no parameter tuning and have optimal regret guarantees under delayed feedback.
On the Initialization for Convex-Concave Min-max Problems
This means that the convergence speed does not have any improvement even if the algorithm starts from the optimal solution, and hence, is oblivious to the initialization.
A closer look at temporal variability in dynamic online learning
Our proposed algorithm is adaptive not only to the temporal variability of the loss functions, but also to the path length of the sequence of comparators when an upper bound is known.
On the Last Iterate Convergence of Momentum Methods
In this paper, we focus on the convergence rate of the last iterate of SGDM.
Parameter-free Stochastic Optimization of Variationally Coherent Functions
We design and analyze an algorithm for first-order stochastic optimization of a large class of functions on $\mathbb{R}^d$.
Adam$^+$: A Stochastic Method with Adaptive Variance Reduction
As a result, Adam$^+$ requires few parameter tuning, as Adam, but it enjoys a provable convergence guarantee.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
A High Probability Analysis of Adaptive SGD with Momentum
We use it to prove for the first time the convergence of the gradients to zero in high probability in the smooth nonconvex setting for Delayed AdaGrad with momentum.
Better Parameter-free Stochastic Optimization with ODE Updates for Coin-Betting
Parameter-free stochastic gradient descent (PFSGD) algorithms do not require setting learning rates while achieving optimal theoretical performance.
Temporal Variability in Implicit Online Learning
We prove a novel static regret bound that depends on the temporal variability of the sequence of loss functions, a quantity which is often encountered when considering dynamic competitors.
A Second look at Exponential and Cosine Step Sizes: Simplicity, Adaptivity, and Performance
Moreover, we show the surprising property that these two strategies are \emph{adaptive} to the noise level in the stochastic gradients of PL functions.
A Modern Introduction to Online Learning
I present first-order and second-order algorithms for online learning with convex losses, in Euclidean and non-Euclidean settings.
Parameter-Free Locally Differentially Private Stochastic Subgradient Descent
We consider the problem of minimizing a convex risk with stochastic subgradients guaranteeing $\epsilon$-locally differentially private ($\epsilon$-LDP).
Kernel Truncated Randomized Ridge Regression: Optimal Rates and Low Noise Acceleration
In this paper, we consider the nonparametric least square regression in a Reproducing Kernel Hilbert Space (RKHS).
Momentum-Based Variance Reduction in Non-Convex SGD
Variance reduction has emerged in recent years as a strong competitor to stochastic gradient descent in non-convex problems, providing the first algorithms to improve upon the converge rate of stochastic gradient descent for finding first-order critical points.
Parameter-Free Online Convex Optimization with Sub-Exponential Noise
We show that BANCO achieves the optimal regret rate in our problem.
Surrogate Losses for Online Learning of Stepsizes in Stochastic Non-Convex Optimization
Stochastic Gradient Descent (SGD) has played a central role in machine learning.
On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizes
In this paper, we start closing this gap: we theoretically analyze in the convex and non-convex settings a generalized version of the AdaGrad stepsizes.
Black-Box Reductions for Parameter-free Online Learning in Banach Spaces
We introduce several new black-box reductions that significantly improve the design of adaptive and parameter-free online learning algorithms by simplifying analysis, improving regret guarantees, and sometimes even improving runtime.
Online Learning for Changing Environments using Coin Betting
A key challenge in online learning is that classical algorithms can be slow to adapt to changing environments.
Efficient Online Bandit Multiclass Learning with O(sqrt{T}) Regret
An efficient bandit algorithm for $\sqrt{T}$-regret in online multiclass prediction?
Training Deep Networks without Learning Rates Through Coin Betting
Instead, we reduce the optimization process to a game of betting on a coin and propose a learning-rate-free optimal algorithm for this scenario.
Ranked #1 on
Stochastic Optimization
on MNIST
Efficient Online Bandit Multiclass Learning with $\tilde{O}(\sqrt{T})$ Regret
The regret bound holds simultaneously with respect to a family of loss functions parameterized by $\eta$, for a range of $\eta$ restricted by the norm of the competitor.
Improved Strongly Adaptive Online Learning using Coin Betting
This paper describes a new parameter-free online learning algorithm for changing environments.
Coin Betting and Parameter-Free Online Learning
We present a new intuitive framework to design parameter-free algorithms for \emph{both} online linear optimization over Hilbert spaces and for learning with expert advice, based on reductions to betting on outcomes of adversarial coins.
High Dimensional Inference with Random Maximum A-Posteriori Perturbations
This paper shows that the expected value of perturb-max inference with low dimensional perturbations can be used sequentially to generate unbiased samples from the Gibbs distribution.
Solving Ridge Regression using Sketched Preconditioned SVRG
We develop a novel preconditioning method for ridge regression, based on recent linear sketching methods.
Scale-Free Online Learning
We design and analyze algorithms for online linear optimization that have optimal regret and at the same time do not need to know any upper or lower bounds on the norm of the loss vectors.
Optimal Non-Asymptotic Lower Bound on the Minimax Regret of Learning with Expert Advice
We prove non-asymptotic lower bounds on the expectation of the maximum of $d$ independent Gaussian variables and the expectation of the maximum of $d$ independent symmetric random walks.
The ABACOC Algorithm: a Novel Approach for Nonparametric Classification of Data Streams
Stream mining poses unique challenges to machine learning: predictive models are required to be scalable, incrementally trainable, must remain bounded in size (even when the data stream is arbitrarily long), and be nonparametric in order to achieve high accuracy even in complex and dynamic environments.
Scale-Free Algorithms for Online Linear Optimization
We design algorithms for online linear optimization that have optimal regret and at the same time do not need to know any upper or lower bounds on the norm of the loss vectors.
A Simple Expression for Mill's Ratio of the Student's $t$-Distribution
I show a simple expression of the Mill's ratio of the Student's t-Distribution.
Fast Rates by Transferring from Auxiliary Hypotheses
In this work we consider the learning setting where, in addition to the training set, the learner receives a collection of auxiliary hypotheses originating from other tasks.
Scalable Greedy Algorithms for Transfer Learning
In this paper we consider the binary transfer learning problem, focusing on how to select and combine sources from a large pool to yield a good performance on a target task.
Simultaneous Model Selection and Optimization through Parameter-free Stochastic Learning
Stochastic gradient descent algorithms for training linear and kernel predictors are gaining more and more importance, thanks to their scalability.
Unconstrained Online Linear Learning in Hilbert Spaces: Minimax Algorithms and Normal Approximations
When $T$ is known, we derive an algorithm with an optimal regret bound (up to constant factors).
Regression-tree Tuning in a Streaming Setting
We consider the problem of maintaining the data-structures of a partition-based regression procedure in a setting where the training data arrives sequentially over time.
Dimension-Free Exponentiated Gradient
We present a new online learning algorithm that extends the exponentiated gradient to infinite dimensional spaces.
On Measure Concentration of Random Maximum A-Posteriori Perturbations
Applying the general result to MAP perturbations can yield a more efficient algorithm to approximate sampling from the Gibbs distribution.
From N to N+1: Multiclass Transfer Incremental Learning
Since the seminal work of Thrun [17], the learning to learn paradigm has been defined as the ability of an agent to improve its performance at each task with experience, with the number of tasks.
A Generalized Online Mirror Descent with Applications to Classification and Regression
A unifying perspective on the design and the analysis of online algorithms is provided by online mirror descent, a general prediction strategy from which most first-order algorithms can be obtained as special cases.
On Multilabel Classification and Ranking with Partial Feedback
We present a novel multilabel/ranking algorithm working in partial information settings.
Learning from Candidate Labeling Sets
In this paper, we propose a semi-supervised framework to model this kind of problems.
New Adaptive Algorithms for Online Classification
We propose a general framework to online learning for classification problems with time-varying potential functions in the adversarial setting.
