1 code implementation • 21 Apr 2023 • Romain Menegaux, Emmanuel Jehanno, Margot Selosse, Julien Mairal
We introduce a novel self-attention mechanism, which we call CSA (Chromatic Self-Attention), which extends the notion of attention scores to attention _filters_, independently modulating the feature channels.
Ranked #1 on Graph Regression on ZINC-500k
no code implementations • NeurIPS 2023 • Michael Arbel, Romain Menegaux, Pierre Wolinski
This work studies the global convergence and implicit bias of Gauss Newton's (GN) when optimizing over-parameterized one-hidden layer networks in the mean-field regime.