Search Results for author:
Found 6 papers, 3 papers with code
Uncovering mesa-optimization algorithms in Transformers
Transformers have become the dominant model in deep learning, but the reason for their superior performance is poorly understood.
Training trajectories, mini-batch losses and the curious role of the learning rate
In particular, for Exponential Moving Average (EMA) and Stochastic Weight Averaging we show that our proposed model matches the observed training trajectories on ImageNet.
Decentralized Learning with Multi-Headed Distillation
We study the effects of data and model architecture heterogeneity and the impact of the underlying communication graph topology on learning efficiency and show that our agents can significantly improve their performance compared to learning in isolation.
Meta-Learning Bidirectional Update Rules
We show that classical gradient-based backpropagation in neural networks can be seen as a special case of a two-state network where one state is used for activations and another for gradients, with update rules derived from the chain rule.
Scale setting the Möbius Domain Wall Fermion on gradient-flowed HISQ action using the omega baryon mass and the gradient-flow scales $t_0$ and $w_0$
We report on a sub-percent scale determination using the omega baryon mass and gradient-flow methods.
High Energy Physics - Lattice High Energy Physics - Phenomenology Nuclear Theory
$F_K / F_π$ from Möbius domain-wall fermions solved on gradient-flowed HISQ ensembles
We report the results of a lattice quantum chromodynamics calculation of $F_K/F_\pi$ using M\"{o}bius domain-wall fermions computed on gradient-flowed $N_f=2+1+1$ highly-improved staggered quark (HISQ) ensembles.
High Energy Physics - Lattice High Energy Physics - Experiment High Energy Physics - Phenomenology Nuclear Theory
