Search Results for author:
Found 19 papers, 6 papers with code
Extra-gradient with player sampling for faster convergence in n-player games
Data-driven modeling increasingly requires to find a Nash equilibrium in multi-player games, e. g. when training GANs.
Q-Probe: A Lightweight Approach to Reward Maximization for Language Models
The idea is to learn a simple linear function on a model's embedding space that can be used to reweight candidate completions.
Repeat After Me: Transformers are Better than State Space Models at Copying
Empirically, we find that transformers outperform GSSMs in terms of efficiency and generalization on synthetic tasks that require copying the context.
Length Generalization in Arithmetic Transformers
We find that relative position embeddings enable length generalization for simple tasks, such as addition: models trained on $5$-digit numbers can perform $15$-digit sums.
Depth Dependence of $μ$P Learning Rates in ReLU MLPs
In this short note we consider random fully connected ReLU networks of width $n$ and depth $L$ equipped with a mean-field weight initialization.
Vision Transformers provably learn spatial structure
On the theoretical side, we consider a binary classification task and show that while the learning problem admits multiple solutions that generalize, our model implicitly learns the spatial structure of the dataset while generalizing: we call this phenomenon patch association.
Dissecting adaptive methods in GANs
By considering an update rule with the magnitude of the Adam update and the normalized direction of SGD, we empirically show that the adaptive magnitude of Adam is key for GAN training.
Towards understanding how momentum improves generalization in deep learning
Stochastic gradient descent (SGD) with momentum is widely used for training modern deep learning architectures.
Adam is no better than normalized SGD: Dissecting how adaptivity improves GAN performance
We empirically show that SGDA with the same vector norm as Adam reaches similar or even better performance than the latter.
Depth separation beyond radial functions
The first contribution of this paper is to extend such results to a more general class of functions, namely functions with piece-wise oscillatory structure, by building on the proof strategy of (Eldan and Shamir, 2016).
Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimization
We introduce MADGRAD, a novel optimization method in the family of AdaGrad adaptive gradient methods.
Dual Averaging is Surprisingly Effective for Deep Learning Optimization
First-order stochastic optimization methods are currently the most widely used class of methods for training deep neural networks.
Auction learning as a two-player game
This not only circumvents the need for an expensive hyper-parameter search (as in prior work), but also provides a principled metric to compare the performance of two auctions (absent from prior work).
A Permutation-Equivariant Neural Network Architecture For Auction Design
Designing an incentive compatible auction that maximizes expected revenue is a central problem in Auction Design.
A mean-field analysis of two-player zero-sum games
Our method identifies mixed equilibria in high dimensions and is demonstrably effective for training mixtures of GANs.
Towards closing the gap between the theory and practice of SVRG
Among the very first variance reduced stochastic methods for solving the empirical risk minimization problem was the SVRG method (Johnson & Zhang 2013).
Extragradient with player sampling for faster Nash equilibrium finding
Data-driven modeling increasingly requires to find a Nash equilibrium in multi-player games, e. g. when training GANs.
Global convergence of neuron birth-death dynamics
Neural networks with a large number of parameters admit a mean-field description, which has recently served as a theoretical explanation for the favorable training properties of "overparameterized" models.
Smoothed analysis of the low-rank approach for smooth semidefinite programs
In order to overcome scalability issues, Burer and Monteiro proposed a factorized approach based on optimizing over a matrix Y of size $n$ by $k$ such that $X = YY^*$ is the SDP variable.
