Search Results for author:
Found 13 papers, 0 papers with code
Towards Understanding Inductive Bias in Transformers: A View From Infinity
We study inductive bias in Transformers in the infinitely over-parameterized Gaussian process limit and argue transformers tend to be biased towards more permutation symmetric functions in sequence space.
Droplets of Good Representations: Grokking as a First Order Phase Transition in Two Layer Networks
A key property of deep neural networks (DNNs) is their ability to learn new features during training.
Speed Limits for Deep Learning
State-of-the-art neural networks require extreme computational power to train.
Spectral-Bias and Kernel-Task Alignment in Physically Informed Neural Networks
Physically informed neural networks (PINNs) are a promising emerging method for solving differential equations.
Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNs
Deep neural networks (DNNs) are powerful tools for compressing and distilling information.
A self consistent theory of Gaussian Processes captures feature learning effects in finite CNNs
Deep neural networks (DNNs) in the infinite width/channel limit have received much attention recently, as they provide a clear analytical window to deep learning via mappings to Gaussian Processes (GPs).
Predicting the Outputs of Finite Networks Trained with Noisy Gradients
A recent line of works studied wide deep neural networks (DNNs) by approximating them as Gaussian Processes (GPs).
Predicting the outputs of finite deep neural networks trained with noisy gradients
A recent line of works studied wide deep neural networks (DNNs) by approximating them as Gaussian Processes (GPs).
Learning Curves for Deep Neural Networks: A field theory perspective
A series of recent works established a rigorous correspondence between very wide deep neural networks (DNNs), trained in a particular manner, and noiseless Bayesian Inference with a certain Gaussian Process (GP) known as the Neural Tangent Kernel (NTK).
Learning Curves for Deep Neural Networks: A Gaussian Field Theory Perspective
In the past decade, deep neural networks (DNNs) came to the fore as the leading machine learning algorithms for a variety of tasks.
The role of a layer in deep neural networks: a Gaussian Process perspective
Leveraging this correspondence, we derive the Deep Gaussian Layer-wise loss functions (DGLs) which, we believe, are the first supervised layer-wise loss functions which are both explicit and competitive in terms of accuracy.
Critical Percolation as a Framework to Analyze the Training of Deep Networks
In this paper we approach two relevant deep learning topics: i) tackling of graph structured input data and ii) a better understanding and analysis of deep networks and related learning algorithms.
Mutual Information, Neural Networks and the Renormalization Group
Physical systems differring in their microscopic details often display strikingly similar behaviour when probed at macroscopic scales.
Disordered Systems and Neural Networks Statistical Mechanics Information Theory Machine Learning Information Theory Machine Learning
