Search Results for author:
Found 7 papers, 4 papers with code
Debiasing a First-order Heuristic for Approximate Bi-level Optimization
Approximate bi-level optimization (ABLO) consists of (outer-level) optimization problems, involving numerical (inner-level) optimization loops.
Sub-Linear Memory: How to Make Performers SLiM
Recent works proposed various linear self-attention mechanisms, scaling only as $O(L)$ for serial computation.
Rethinking Attention with Performers
We introduce Performers, Transformer architectures which can estimate regular (softmax) full-rank-attention Transformers with provable accuracy, but using only linear (as opposed to quadratic) space and time complexity, without relying on any priors such as sparsity or low-rankness.
Ranked #7 on
Offline RL
on D4RL
Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers
In response, solutions that exploit the structure and sparsity of the learned attention matrix have blossomed.
UFO-BLO: Unbiased First-Order Bilevel Optimization
Bilevel optimization (BLO) is a popular approach with many applications including hyperparameter optimization, neural architecture search, adversarial robustness and model-agnostic meta-learning.
CWY Parametrization: a Solution for Parallelized Optimization of Orthogonal and Stiefel Matrices
We introduce an efficient approach for optimization over orthogonal groups on highly parallel computation units such as GPUs or TPUs.
Stochastic Flows and Geometric Optimization on the Orthogonal Group
We present a new class of stochastic, geometrically-driven optimization algorithms on the orthogonal group $O(d)$ and naturally reductive homogeneous manifolds obtained from the action of the rotation group $SO(d)$.
