Search Results for author:
Found 89 papers, 8 papers with code
Depth Separation in Norm-Bounded Infinite-Width Neural Networks
We also show that a similar statement in the reverse direction is not possible: any function learnable with polynomial sample complexity by a norm-controlled depth-2 ReLU network with infinite width is also learnable with polynomial sample complexity by a norm-controlled depth-3 ReLU network.
Generalization in Kernel Regression Under Realistic Assumptions
It is by now well-established that modern over-parameterized models seem to elude the bias-variance tradeoff and generalize well despite overfitting noise.
An Algorithm with Optimal Dimension-Dependence for Zero-Order Nonsmooth Nonconvex Stochastic Optimization
Recent works proposed several stochastic zero-order algorithms that solve this task, all of which suffer from a dimension-dependence of $\Omega(d^{3/2})$ where $d$ is the dimension of the problem, which was conjectured to be optimal.
From Tempered to Benign Overfitting in ReLU Neural Networks
Thus, we show that the input dimension has a crucial role on the type of overfitting in this setting, which we also validate empirically for intermediate dimensions.
Deterministic Nonsmooth Nonconvex Optimization
In particular, we prove a lower bound of $\Omega(d)$ for any deterministic algorithm.
On the Complexity of Finding Small Subgradients in Nonsmooth Optimization
We study the oracle complexity of producing $(\delta,\epsilon)$-stationary points of Lipschitz functions, in the sense proposed by Zhang et al. [2020].
Reconstructing Training Data from Trained Neural Networks
We propose a novel reconstruction scheme that stems from recent theoretical results about the implicit bias in training neural networks with gradient-based methods.
The Sample Complexity of One-Hidden-Layer Neural Networks
We study norm-based uniform convergence bounds for neural networks, aiming at a tight understanding of how these are affected by the architecture and type of norm constraint, for the simple class of scalar-valued one-hidden-layer networks, and inputs bounded in Euclidean norm.
Gradient Methods Provably Converge to Non-Robust Networks
Despite a great deal of research, it is still unclear why neural networks are so susceptible to adversarial examples.
Width is Less Important than Depth in ReLU Neural Networks
We solve an open question from Lu et al. (2017), by showing that any target network with inputs in $\mathbb{R}^d$ can be approximated by a width $O(d)$ network (independent of the target network's architecture), whose number of parameters is essentially larger only by a linear factor.
Implicit Regularization Towards Rank Minimization in ReLU Networks
We study the conjectured relationship between the implicit regularization in neural networks, trained with gradient-based methods, and rank minimization of their weight matrices.
The Implicit Bias of Benign Overfitting
In this paper, we provide several new results on when one can or cannot expect benign overfitting to occur, for both regression and classification tasks.
Convergence Results For Q-Learning With Experience Replay
A commonly used heuristic in RL is experience replay (e. g.~\citet{lin1993reinforcement, mnih2015human}), in which a learner stores and re-uses past trajectories as if they were sampled online.
Replay For Safety
Experience replay \citep{lin1993reinforcement, mnih2015human} is a widely used technique to achieve efficient use of data and improved performance in RL algorithms.
On the Optimal Memorization Power of ReLU Neural Networks
We prove that having such a large bit complexity is both necessary and sufficient for memorization with a sub-linear number of parameters.
A Stochastic Newton Algorithm for Distributed Convex Optimization
We propose and analyze a stochastic Newton algorithm for homogeneous distributed stochastic convex optimization, where each machine can calculate stochastic gradients of the same population objective, as well as stochastic Hessian-vector products (products of an independent unbiased estimator of the Hessian of the population objective with arbitrary vectors), with many such stochastic computations performed between rounds of communication.
On Margin Maximization in Linear and ReLU Networks
The implicit bias of neural networks has been extensively studied in recent years.
Random Shuffling Beats SGD Only After Many Epochs on Ill-Conditioned Problems
Perhaps surprisingly, we prove that when the condition number is taken into account, without-replacement SGD \emph{does not} significantly improve on with-replacement SGD in terms of worst-case bounds, unless the number of epochs (passes over the data) is larger than the condition number.
Learning a Single Neuron with Bias Using Gradient Descent
We theoretically study the fundamental problem of learning a single neuron with a bias term ($\mathbf{x} \mapsto \sigma(<\mathbf{w},\mathbf{x}> + b)$) in the realizable setting with the ReLU activation, using gradient descent.
Oracle Complexity in Nonsmooth Nonconvex Optimization
For this approach, we prove under a mild assumption an inherent trade-off between oracle complexity and smoothness: On the one hand, smoothing a nonsmooth nonconvex function can be done very efficiently (e. g., by randomized smoothing), but with dimension-dependent factors in the smoothness parameter, which can strongly affect iteration complexity when plugging into standard smooth optimization methods.
The Min-Max Complexity of Distributed Stochastic Convex Optimization with Intermittent Communication
We resolve the min-max complexity of distributed stochastic convex optimization (up to a log factor) in the intermittent communication setting, where $M$ machines work in parallel over the course of $R$ rounds of communication to optimize the objective, and during each round of communication, each machine may sequentially compute $K$ stochastic gradient estimates.
The Connection Between Approximation, Depth Separation and Learnability in Neural Networks
On the other hand, the fact that deep networks can efficiently express a target function does not mean that this target function can be learned efficiently by deep neural networks.
Size and Depth Separation in Approximating Benign Functions with Neural Networks
We show a complexity-theoretic barrier to proving such results beyond size $O(d\log^2(d))$, but also show an explicit benign function, that can be approximated with networks of size $O(d)$ and not with networks of size $o(d/\log d)$.
Implicit Regularization in ReLU Networks with the Square Loss
For one hidden-layer networks, we prove a similar result, where in general it is impossible to characterize implicit regularization properties in this manner, except for the "balancedness" property identified in Du et al. [2018].
High-Order Oracle Complexity of Smooth and Strongly Convex Optimization
In this note, we consider the complexity of optimizing a highly smooth (Lipschitz $k$-th order derivative) and strongly convex function, via calls to a $k$-th order oracle which returns the value and first $k$ derivatives of the function at a given point, and where the dimension is unrestricted.
Gradient Methods Never Overfit On Separable Data
A line of recent works established that when training linear predictors over separable data, using gradient methods and exponentially-tailed losses, the predictors asymptotically converge in direction to the max-margin predictor.
The Effects of Mild Over-parameterization on the Optimization Landscape of Shallow ReLU Neural Networks
We prove that while the objective is strongly convex around the global minima when the teacher and student networks possess the same number of neurons, it is not even \emph{locally convex} after any amount of over-parameterization.
Neural Networks with Small Weights and Depth-Separation Barriers
To show this, we study a seemingly unrelated problem of independent interest: Namely, whether there are polynomially-bounded functions which require super-polynomial weights in order to approximate with constant-depth neural networks.
Can We Find Near-Approximately-Stationary Points of Nonsmooth Nonconvex Functions?
It is well-known that given a bounded, smooth nonconvex function, standard gradient-based methods can find $\epsilon$-stationary points (where the gradient norm is less than $\epsilon$) in $\mathcal{O}(1/\epsilon^2)$ iterations.
Is Local SGD Better than Minibatch SGD?
We study local SGD (also known as parallel SGD and federated averaging), a natural and frequently used stochastic distributed optimization method.
Proving the Lottery Ticket Hypothesis: Pruning is All You Need
The lottery ticket hypothesis (Frankle and Carbin, 2018), states that a randomly-initialized network contains a small subnetwork such that, when trained in isolation, can compete with the performance of the original network.
Learning a Single Neuron with Gradient Methods
We consider the fundamental problem of learning a single neuron $x \mapsto\sigma(w^\top x)$ using standard gradient methods.
The Complexity of Finding Stationary Points with Stochastic Gradient Descent
We study the iteration complexity of stochastic gradient descent (SGD) for minimizing the gradient norm of smooth, possibly nonconvex functions.
How Good is SGD with Random Shuffling?
In contrast to the majority of existing theoretical works, which assume that individual functions are sampled with replacement, we focus here on popular but poorly-understood heuristics, which involve going over random permutations of the individual functions.
Depth Separations in Neural Networks: What is Actually Being Separated?
Existing depth separation results for constant-depth networks essentially show that certain radial functions in $\mathbb{R}^d$, which can be easily approximated with depth $3$ networks, cannot be approximated by depth $2$ networks, even up to constant accuracy, unless their size is exponential in $d$.
On the Power and Limitations of Random Features for Understanding Neural Networks
Recently, a spate of papers have provided positive theoretical results for training over-parameterized neural networks (where the network size is larger than what is needed to achieve low error).
The Complexity of Making the Gradient Small in Stochastic Convex Optimization
Notably, we show that in the global oracle/statistical learning model, only logarithmic dependence on smoothness is required to find a near-stationary point, whereas polynomial dependence on smoothness is necessary in the local stochastic oracle model.
Space lower bounds for linear prediction in the streaming model
We show that fundamental learning tasks, such as finding an approximate linear separator or linear regression, require memory at least \emph{quadratic} in the dimension, in a natural streaming setting.
Global Non-convex Optimization with Discretized Diffusions
An Euler discretization of the Langevin diffusion is known to converge to the global minimizers of certain convex and non-convex optimization problems.
Exponential Convergence Time of Gradient Descent for One-Dimensional Deep Linear Neural Networks
We study the dynamics of gradient descent on objective functions of the form $f(\prod_{i=1}^{k} w_i)$ (with respect to scalar parameters $w_1,\ldots, w_k$), which arise in the context of training depth-$k$ linear neural networks.
A Tight Convergence Analysis for Stochastic Gradient Descent with Delayed Updates
We provide tight finite-time convergence bounds for gradient descent and stochastic gradient descent on quadratic functions, when the gradients are delayed and reflect iterates from $\tau$ rounds ago.
Are ResNets Provably Better than Linear Predictors?
In this paper, we rigorously prove that arbitrarily deep, nonlinear residual units indeed exhibit this behavior, in the sense that the optimization landscape contains no local minima with value above what can be obtained with a linear predictor (namely a 1-layer network).
Detecting Correlations with Little Memory and Communication
We study the problem of identifying correlations in multivariate data, under information constraints: Either on the amount of memory that can be used by the algorithm, or the amount of communication when the data is distributed across several machines.
Spurious Local Minima are Common in Two-Layer ReLU Neural Networks
We consider the optimization problem associated with training simple ReLU neural networks of the form $\mathbf{x}\mapsto \sum_{i=1}^{k}\max\{0,\mathbf{w}_i^\top \mathbf{x}\}$ with respect to the squared loss.
Size-Independent Sample Complexity of Neural Networks
We study the sample complexity of learning neural networks, by providing new bounds on their Rademacher complexity assuming norm constraints on the parameter matrix of each layer.
Weight Sharing is Crucial to Succesful Optimization
Exploiting the great expressive power of Deep Neural Network architectures, relies on the ability to train them.
Bandit Regret Scaling with the Effective Loss Range
We study how the regret guarantees of nonstochastic multi-armed bandits can be improved, if the effective range of the losses in each round is small (e. g. the maximal difference between two losses in a given round).
Failures of Gradient-Based Deep Learning
In recent years, Deep Learning has become the go-to solution for a broad range of applications, often outperforming state-of-the-art.
Online Learning with Local Permutations and Delayed Feedback
In this paper, we consider the applicability of this setting to convex online learning with delayed feedback, in which the feedback on the prediction made in round $t$ arrives with some delay $\tau$.
Communication-efficient Algorithms for Distributed Stochastic Principal Component Analysis
We study algorithms for estimating the leading principal component of the population covariance matrix that are both communication-efficient and achieve estimation error of the order of the centralized ERM solution that uses all $mn$ samples.
Without-Replacement Sampling for Stochastic Gradient Methods
Stochastic gradient methods for machine learning and optimization problems are usually analyzed assuming data points are sampled *with* replacement.
Oracle Complexity of Second-Order Methods for Finite-Sum Problems
Finite-sum optimization problems are ubiquitous in machine learning, and are commonly solved using first-order methods which rely on gradient computations.
Depth-Width Tradeoffs in Approximating Natural Functions with Neural Networks
We provide several new depth-based separation results for feed-forward neural networks, proving that various types of simple and natural functions can be better approximated using deeper networks than shallower ones, even if the shallower networks are much larger.
Distribution-Specific Hardness of Learning Neural Networks
Although neural networks are routinely and successfully trained in practice using simple gradient-based methods, most existing theoretical results are negative, showing that learning such networks is difficult, in a worst-case sense over all data distributions.
Dimension-Free Iteration Complexity of Finite Sum Optimization Problems
Many canonical machine learning problems boil down to a convex optimization problem with a finite sum structure.
On the Iteration Complexity of Oblivious First-Order Optimization Algorithms
We consider a broad class of first-order optimization algorithms which are \emph{oblivious}, in the sense that their step sizes are scheduled regardless of the function under consideration, except for limited side-information such as smoothness or strong convexity parameters.
Without-Replacement Sampling for Stochastic Gradient Methods: Convergence Results and Application to Distributed Optimization
Stochastic gradient methods for machine learning and optimization problems are usually analyzed assuming data points are sampled \emph{with} replacement.
The Power of Depth for Feedforward Neural Networks
We show that there is a simple (approximately radial) function on $\reals^d$, expressible by a small 3-layer feedforward neural networks, which cannot be approximated by any 2-layer network, to more than a certain constant accuracy, unless its width is exponential in the dimension.
Multi-Player Bandits -- a Musical Chairs Approach
We consider a variant of the stochastic multi-armed bandit problem, where multiple players simultaneously choose from the same set of arms and may collide, receiving no reward.
On the Quality of the Initial Basin in Overspecified Neural Networks
Deep learning, in the form of artificial neural networks, has achieved remarkable practical success in recent years, for a variety of difficult machine learning applications.
Convergence of Stochastic Gradient Descent for PCA
We consider the problem of principal component analysis (PCA) in a streaming stochastic setting, where our goal is to find a direction of approximate maximal variance, based on a stream of i. i. d.
Fast Stochastic Algorithms for SVD and PCA: Convergence Properties and Convexity
We study the convergence properties of the VR-PCA algorithm introduced by \cite{shamir2015stochastic} for fast computation of leading singular vectors.
An Optimal Algorithm for Bandit and Zero-Order Convex Optimization with Two-Point Feedback
We consider the closely related problems of bandit convex optimization with two-point feedback, and zero-order stochastic convex optimization with two function evaluations per round.
Communication Complexity of Distributed Convex Learning and Optimization
We study the fundamental limits to communication-efficient distributed methods for convex learning and optimization, under different assumptions on the information available to individual machines, and the types of functions considered.
On Lower and Upper Bounds for Smooth and Strongly Convex Optimization Problems
This, in turn, reveals a powerful connection between a class of optimization algorithms and the analytic theory of polynomials whereby new lower and upper bounds are derived.
On the Complexity of Learning with Kernels
In this paper, we study lower bounds on the error attainable by such methods as a function of the number of entries observed in the kernel matrix or the rank of an approximate kernel matrix.
Attribute Efficient Linear Regression with Data-Dependent Sampling
In this paper we analyze a budgeted learning setting, in which the learner can only choose and observe a small subset of the attributes of each training example.
On the Computational Efficiency of Training Neural Networks
It is well-known that neural networks are computationally hard to train.
Nonstochastic Multi-Armed Bandits with Graph-Structured Feedback
This naturally models several situations where the losses of different actions are related, and knowing the loss of one action provides information on the loss of other actions.
A Stochastic PCA and SVD Algorithm with an Exponential Convergence Rate
We describe and analyze a simple algorithm for principal component analysis and singular value decomposition, VR-PCA, which uses computationally cheap stochastic iterations, yet converges exponentially fast to the optimal solution.
On the Complexity of Bandit Linear Optimization
We study the attainable regret for online linear optimization problems with bandit feedback, where unlike the full-information setting, the player can only observe its own loss rather than the full loss vector.
The Sample Complexity of Learning Linear Predictors with the Squared Loss
In this short note, we provide a sample complexity lower bound for learning linear predictors with respect to the squared loss.
Graph Approximation and Clustering on a Budget
We consider the problem of learning from a similarity matrix (such as spectral clustering and lowd imensional embedding), when computing pairwise similarities are costly, and only a limited number of entries can be observed.
Communication Efficient Distributed Optimization using an Approximate Newton-type Method
We present a novel Newton-type method for distributed optimization, which is particularly well suited for stochastic optimization and learning problems.
Online Learning with Costly Features and Labels
In particular, we show that with switching costs, the attainable rate with bandit feedback is $T^{2/3}$.
Fundamental Limits of Online and Distributed Algorithms for Statistical Learning and Estimation
Many machine learning approaches are characterized by information constraints on how they interact with the training data.
Probabilistic Label Trees for Efficient Large Scale Image Classification
Large-scale recognition problems with thousands of classes pose a particular challenge because applying the classifier requires more computation as the number of classes grows.
An Algorithm for Training Polynomial Networks
The main goal of this paper is the derivation of an efficient layer-by-layer algorithm for training such networks, which we denote as the \emph{Basis Learner}.
Online Learning with Switching Costs and Other Adaptive Adversaries
In particular, we show that with switching costs, the attainable rate with bandit feedback is $\widetilde{\Theta}(T^{2/3})$.
Relax and Randomize : From Value to Algorithms
We show a principled way of deriving online learning algorithms from a minimax analysis.
On the Complexity of Bandit and Derivative-Free Stochastic Convex Optimization
The problem of stochastic convex optimization with bandit feedback (in the learning community) or without knowledge of gradients (in the optimization community) has received much attention in recent years, in the form of algorithms and performance upper bounds.
Learning with the weighted trace-norm under arbitrary sampling distributions
We provide rigorous guarantees on learning with the weighted trace-norm under arbitrary sampling distributions.
From Bandits to Experts: On the Value of Side-Observations
We consider an adversarial online learning setting where a decision maker can choose an action in every stage of the game.
Efficient Online Learning via Randomized Rounding
Most online algorithms used in machine learning today are based on variants of mirror descent or follow-the-leader.
Better Mini-Batch Algorithms via Accelerated Gradient Methods
Mini-batch algorithms have recently received significant attention as a way to speed-up stochastic convex optimization problems.
Efficient Learning of Generalized Linear and Single Index Models with Isotonic Regression
In this paper, we provide algorithms for learning GLMs and SIMs, which are both computationally and statistically efficient.
Efficient Transductive Online Learning via Randomized Rounding
Most traditional online learning algorithms are based on variants of mirror descent or follow-the-leader.
Learning Exponential Families in High-Dimensions: Strong Convexity and Sparsity
The versatility of exponential families, along with their attendant convexity properties, make them a popular and effective statistical model.
On the Reliability of Clustering Stability in the Large Sample Regime
In this paper, we provide a set of general sufficient conditions, which ensure the reliability of clustering stability estimators in the large sample regime.
