Search Results for author:
Found 7 papers, 2 papers with code
Forward Learning with Top-Down Feedback: Empirical and Analytical Characterization
"Forward-only" algorithms, which train neural networks while avoiding a backward pass, have recently gained attention as a way of solving the biologically unrealistic aspects of backpropagation.
Rigorous dynamical mean field theory for stochastic gradient descent methods
We prove closed-form equations for the exact high-dimensional asymptotics of a family of first order gradient-based methods, learning an estimator (e. g. M-estimator, shallow neural network, ...) from observations on Gaussian data with empirical risk minimization.
Learning curves for the multi-class teacher-student perceptron
For Gaussian teacher weights, we investigate the performance of ERM with both cross-entropy and square losses, and explore the role of ridge regularisation in approaching Bayes-optimality.
The effective noise of Stochastic Gradient Descent
In the under-parametrized regime, where the final training error is positive, the SGD dynamics reaches a stationary state and we define an effective temperature from the fluctuation-dissipation theorem, computed from dynamical mean-field theory.
Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problem
In this paper we investigate how gradient-based algorithms such as gradient descent, (multi-pass) stochastic gradient descent, its persistent variant, and the Langevin algorithm navigate non-convex loss-landscapes and which of them is able to reach the best generalization error at limited sample complexity.
Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification
We define a particular stochastic process for which SGD can be extended to a continuous-time limit that we call stochastic gradient flow.
The role of regularization in classification of high-dimensional noisy Gaussian mixture
We also illustrate the interpolation peak at low regularization, and analyze the role of the respective sizes of the two clusters.
