Search Results for author:
Found 29 papers, 8 papers with code
The Implicit Bias of Gradient Noise: A Symmetry Perspective
For one class of symmetry, SGD naturally converges to solutions that have a balanced and aligned gradient noise.
When Does Feature Learning Happen? Perspective from an Analytically Solvable Model
We identify and solve a hidden-layer model that is analytically tractable at any finite width and whose limits exhibit both the kernel phase and the feature learning phase.
Symmetry Leads to Structured Constraint of Learning
Due to common architecture designs, symmetries exist extensively in contemporary neural networks.
Law of Balance and Stationary Distribution of Stochastic Gradient Descent
The stochastic gradient descent (SGD) algorithm is the algorithm we use to train neural networks.
On the Stepwise Nature of Self-Supervised Learning
We present a simple picture of the training process of joint embedding self-supervised learning methods.
The Probabilistic Stability of Stochastic Gradient Descent
Characterizing and understanding the stability of Stochastic Gradient Descent (SGD) remains an open problem in deep learning.
spred: Solving $L_1$ Penalty with SGD
We propose to minimize a generic differentiable objective with $L_1$ constraint using a simple reparametrization and straightforward stochastic gradient descent.
What shapes the loss landscape of self-supervised learning?
Prevention of complete and dimensional collapse of representations has recently become a design principle for self-supervised learning (SSL).
Exact Phase Transitions in Deep Learning
This work reports deep-learning-unique first-order and second-order phase transitions, whose phenomenology closely follows that in statistical physics.
Posterior Collapse of a Linear Latent Variable Model
This work identifies the existence and cause of a type of posterior collapse that frequently occurs in the Bayesian deep learning practice.
Exact Solutions of a Deep Linear Network
This work finds the analytical expression of the global minima of a deep linear network with weight decay and stochastic neurons, a fundamental model for understanding the landscape of neural networks.
Stochastic Neural Networks with Infinite Width are Deterministic
This work theoretically studies stochastic neural networks, a main type of neural network in use.
Logarithmic landscape and power-law escape rate of SGD
Stochastic gradient descent (SGD) undergoes complicated multiplicative noise for the mean-square loss.
SGD Can Converge to Local Maxima
Stochastic gradient descent (SGD) is widely used for the nonlinear, nonconvex problem of training deep neural networks, but its behavior remains poorly understood.
SGD with a Constant Large Learning Rate Can Converge to Local Maxima
Previous works on stochastic gradient descent (SGD) often focus on its success.
Theoretically Motivated Data Augmentation and Regularization for Portfolio Construction
The task we consider is portfolio construction in a speculative market, a fundamental problem in modern finance.
Power-law escape rate of SGD
Stochastic gradient descent (SGD) undergoes complicated multiplicative noise for the mean-square loss.
On the Distributional Properties of Adaptive Gradients
Adaptive gradient methods have achieved remarkable success in training deep neural networks on a wide variety of tasks.
Strength of Minibatch Noise in SGD
The noise in stochastic gradient descent (SGD), caused by minibatch sampling, is poorly understood despite its practical importance in deep learning.
Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descent
In the vanishing learning rate regime, stochastic gradient descent (SGD) is now relatively well understood.
Cross-Modal Generalization: Learning in Low Resource Modalities via Meta-Alignment
In this work, we propose algorithms for cross-modal generalization: a learning paradigm to train a model that can (1) quickly perform new tasks in a target modality (i. e. meta-learning) and (2) doing so while being trained on a different source modality.
An Investigation of how Label Smoothing Affects Generalization
In this paper, as a step towards understanding why label smoothing is effective, we propose a theoretical framework to show how label smoothing provides in controlling the generalization loss.
Neural Networks Fail to Learn Periodic Functions and How to Fix It
Previous literature offers limited clues on how to learn a periodic function using modern neural networks.
Volumization as a Natural Generalization of Weight Decay
We propose a novel regularization method, called \textit{volumization}, for neural networks.
Learning Not to Learn in the Presence of Noisy Labels
Learning in the presence of label noise is a challenging yet important task: it is crucial to design models that are robust in the presence of mislabeled datasets.
LaProp: Separating Momentum and Adaptivity in Adam
We also bound the regret of Laprop on a convex problem and show that our bound differs from that of Adam by a key factor, which demonstrates its advantage.
Think Locally, Act Globally: Federated Learning with Local and Global Representations
To this end, we propose a new federated learning algorithm that jointly learns compact local representations on each device and a global model across all devices.
A Simple Approach to the Noisy Label Problem Through the Gambler's Loss
Learning in the presence of label noise is a challenging yet important task.
Deep Gamblers: Learning to Abstain with Portfolio Theory
We deal with the \textit{selective classification} problem (supervised-learning problem with a rejection option), where we want to achieve the best performance at a certain level of coverage of the data.
