Search Results for author:
Found 14 papers, 5 papers with code
Linear Transformers are Versatile In-Context Learners
Recent research has demonstrated that transformers, particularly linear attention models, implicitly execute gradient-descent-like algorithms on data provided in-context during their forward inference step.
Uncovering mesa-optimization algorithms in Transformers
Transformers have become the dominant model in deep learning, but the reason for their superior performance is poorly understood.
Continual Few-Shot Learning Using HyperTransformers
We focus on the problem of learning without forgetting from multiple tasks arriving sequentially, where each task is defined using a few-shot episode of novel or already seen classes.
Training trajectories, mini-batch losses and the curious role of the learning rate
In particular, for Exponential Moving Average (EMA) and Stochastic Weight Averaging we show that our proposed model matches the observed training trajectories on ImageNet.
Transformers learn in-context by gradient descent
We start by providing a simple weight construction that shows the equivalence of data transformations induced by 1) a single linear self-attention layer and by 2) gradient-descent (GD) on a regression loss.
Decentralized Learning with Multi-Headed Distillation
We study the effects of data and model architecture heterogeneity and the impact of the underlying communication graph topology on learning efficiency and show that our agents can significantly improve their performance compared to learning in isolation.
Fine-tuning Image Transformers using Learnable Memory
In this paper we propose augmenting Vision Transformer models with learnable memory tokens.
GradMax: Growing Neural Networks using Gradient Information
The architecture and the parameters of neural networks are often optimized independently, which requires costly retraining of the parameters whenever the architecture is modified.
HyperTransformer: Model Generation for Supervised and Semi-Supervised Few-Shot Learning
In this work we propose a HyperTransformer, a Transformer-based model for supervised and semi-supervised few-shot learning that generates weights of a convolutional neural network (CNN) directly from support samples.
HyperTransformer: Attention-Based CNN Model Generation from Few Samples
In this work we propose a HyperTransformer, a transformer based model that generates all weights of a CNN model directly from the support samples.
Meta-Learning Bidirectional Update Rules
We show that classical gradient-based backpropagation in neural networks can be seen as a special case of a two-state network where one state is used for activations and another for gradients, with update rules derived from the chain rule.
Underspecification Presents Challenges for Credibility in Modern Machine Learning
Predictors returned by underspecified pipelines are often treated as equivalent based on their training domain performance, but we show here that such predictors can behave very differently in deployment domains.
No Pressure! Addressing the Problem of Local Minima in Manifold Learning Algorithms
However, due to a complicated nonconvex objective function, these methods can easily get stuck in local minima and their embedding quality can be poor.
A fast, universal algorithm to learn parametric nonlinear embeddings
This has two advantages: 1) The algorithm is universal in that a specific learning algorithm for any choice of embedding and mapping can be constructed by simply reusing existing algorithms for the embedding and for the mapping.
