Search Results for author:
Found 17 papers, 9 papers with code
A Dynamical Model of Neural Scaling Laws
On a variety of tasks, the performance of neural networks predictably improves with training time, dataset size and model size across many orders of magnitude.
Grokking as the Transition from Lazy to Rich Training Dynamics
We identify sufficient statistics for the test loss of such a network, and tracking these over training reveals that grokking arises in this setting when the network first attempts to fit a kernel regression solution with its initial features, followed by late-time feature learning where a generalizing solution is identified after train loss is already low.
Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit
We provide experiments demonstrating that residual architectures including convolutional ResNets and Vision Transformers trained with this parameterization exhibit transfer of optimal hyperparameters across width and depth on CIFAR-10 and ImageNet.
Loss Dynamics of Temporal Difference Reinforcement Learning
We study how learning dynamics and plateaus depend on feature structure, learning rate, discount factor, and reward function.
Feature-Learning Networks Are Consistent Across Widths At Realistic Scales
We call this the bias of narrower width.
Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networks
However, in the rich, feature learning regime, the fluctuations of the kernels and predictions are dynamically coupled with a variance that can be computed self-consistently.
The Onset of Variance-Limited Behavior for Networks in the Lazy and Rich Regimes
For small training set sizes $P$, the generalization error of wide neural networks is well-approximated by the error of an infinite width neural network (NN), either in the kernel or mean-field/feature-learning regime.
The Influence of Learning Rule on Representation Dynamics in Wide Neural Networks
In the lazy limit, we find that DFA and Hebb can only learn using the last layer features, while full FA can utilize earlier layers with a scale determined by the initial correlation between feedforward and feedback weight matrices.
Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networks
We analyze feature learning in infinite-width neural networks trained with gradient flow through a self-consistent dynamical field theory.
Neural Networks as Kernel Learners: The Silent Alignment Effect
Can neural networks in the rich feature learning regime learn a kernel machine with a data-dependent kernel?
Capacity of Group-invariant Linear Readouts from Equivariant Representations: How Many Objects can be Linearly Classified Under All Possible Views?
We find that the fraction of separable dichotomies is determined by the dimension of the space that is fixed by the group action.
Learning Curves for SGD on Structured Features
To analyze the influence of data structure on test loss dynamics, we study an exactly solveable model of stochastic gradient descent (SGD) on mean square loss which predicts test loss when training on features with arbitrary covariance structure.
Out-of-Distribution Generalization in Kernel Regression
Here, we study generalization in kernel regression when the training and test distributions are different using methods from statistical physics.
BIG-bench Machine Learning
Out-of-Distribution Generalization
+1
A Theory of Neural Tangent Kernel Alignment and Its Influence on Training
In this work, we seek to theoretically understand kernel alignment, a prominent and ubiquitous structural change that aligns the NTK with the target function.
Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networks
We present applications of our theory to real and synthetic datasets, and for many kernels including those that arise from training deep neural networks in the infinite-width limit.
Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks
We derive analytical expressions for the generalization performance of kernel regression as a function of the number of training samples using theoretical methods from Gaussian processes and statistical physics.
Pre-Synaptic Pool Modification (PSPM): A Supervised Learning Procedure for Spiking Neural Networks
Learning synaptic weights of spiking neural network (SNN) models that can reproduce target spike trains from provided neural firing data is a central problem in computational neuroscience and spike-based computing.
