Search Results for author:
Found 4 papers, 2 papers with code
Why Do We Need Weight Decay in Modern Deep Learning?
In this work, we highlight that the role of weight decay in modern deep learning is different from its regularization effect studied in classical learning theory.
SGD with Large Step Sizes Learns Sparse Features
We present empirical observations that commonly used large step sizes (i) lead the iterates to jump from one side of a valley to the other causing loss stabilization, and (ii) this stabilization induces a hidden stochastic dynamics orthogonal to the bouncing directions that biases it implicitly toward sparse predictors.
Accelerated SGD for Non-Strongly-Convex Least Squares
We consider stochastic approximation for the least squares regression problem in the non-strongly convex setting.
Last iterate convergence of SGD for Least-Squares in the Interpolation regime
Motivated by the recent successes of neural networks that have the ability to fit the data perfectly and generalize well, we study the noiseless model in the fundamental least-squares setup.
