Search Results for author:
Found 10 papers, 3 papers with code
Hallmarks of Optimization Trajectories in Neural Networks and LLMs: The Lengths, Bends, and Dead Ends
We propose a fresh take on understanding the mechanisms of neural networks by analyzing the rich structure of parameters contained within their optimization trajectories.
Recurrent Distance Filtering for Graph Representation Learning
Graph neural networks based on iterative one-hop message passing have been shown to struggle in harnessing the information from distant nodes effectively.
Simplifying Transformer Blocks
A simple design recipe for deep Transformers is to compose identical building blocks.
The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limit
Motivated by the success of Transformers, we study the covariance matrix of a modified Softmax-based attention model with skip connections in the proportional limit of infinite-depth-and-width.
Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagation
Skip connections and normalisation layers form two standard architectural components that are ubiquitous for the training of Deep Neural Networks (DNNs), but whose precise roles are poorly understood.
UncertaINR: Uncertainty Quantification of End-to-End Implicit Neural Representations for Computed Tomography
To that end, we study a Bayesian reformulation of INRs, UncertaINR, in the context of computed tomography, and evaluate several Bayesian deep learning implementations in terms of accuracy and calibration.
Probabilistic fine-tuning of pruning masks and PAC-Bayes self-bounded learning
In the linear model, we show that a PAC-Bayes generalization error bound is controlled by the magnitude of the change in feature alignment between the 'prior' and 'posterior' data.
Feature Kernel Distillation
Trained Neural Networks (NNs) can be viewed as data-dependent kernel machines, with predictions determined by the inner product of last-layer representations across inputs, referred to as the feature kernel.
Bayesian Deep Ensembles via the Neural Tangent Kernel
We explore the link between deep ensembles and Gaussian processes (GPs) through the lens of the Neural Tangent Kernel (NTK): a recent development in understanding the training dynamics of wide neural networks (NNs).
