Search Results for author:
Found 105 papers, 42 papers with code
Extra-gradient with player sampling for faster convergence in n-player games
Data-driven modeling increasingly requires to find a Nash equilibrium in multi-player games, e. g. when training GANs.
Computational-Statistical Gaps in Gaussian Single-Index Models
Single-Index Models are high-dimensional regression problems with planted structure, whereby labels depend on an unknown one-dimensional projection of the input via a generic, non-linear, and potentially non-deterministic transformation.
Stochastic Optimal Control Matching
Our work introduces Stochastic Optimal Control Matching (SOCM), a novel Iterative Diffusion Optimization (IDO) technique for stochastic optimal control that stems from the same philosophy as the conditional score matching loss for diffusion models.
On Learning Gaussian Multi-index Models with Gradient Flow
We study gradient flow on the multi-index regression problem for high-dimensional Gaussian data.
Symmetric Single Index Learning
Learning this model with SGD is relatively well-understood, whereby the so-called information exponent of the link function governs a polynomial sample complexity rate.
On Single Index Models beyond Gaussian Data
Sparse high-dimensional functions have arisen as a rich framework to study the behavior of gradient-descent methods using shallow neural networks, showcasing their ability to perform feature learning beyond linear models.
A Neural Collapse Perspective on Feature Evolution in Graph Neural Networks
We start with an empirical study that shows that a decrease in within-class variability is also prevalent in the node-wise classification setting, however, not to the extent observed in the instance-wise case.
Conditionally Strongly Log-Concave Generative Models
There is a growing gap between the impressive results of deep image generative models and classical algorithms that offer theoretical guarantees.
Data-driven multiscale modeling of subgrid parameterizations in climate models
Subgrid parameterizations, which represent physical processes occurring below the resolution of current climate models, are an important component in producing accurate, long-term predictions for the climate.
A Functional-Space Mean-Field Theory of Partially-Trained Three-Layer Neural Networks
To understand the training dynamics of neural networks (NNs), prior studies have considered the infinite-width mean-field (MF) limit of two-layer NN, establishing theoretical guarantees of its convergence under gradient flow training as well as its approximation and generalization capabilities.
Learning Single-Index Models with Shallow Neural Networks
Single-index models are a class of functions given by an unknown univariate ``link'' function applied to an unknown one-dimensional projection of the input.
Towards Antisymmetric Neural Ansatz Separation
We study separations between two fundamental models (or \emph{Ans\"atze}) of antisymmetric functions, that is, functions $f$ of the form $f(x_{\sigma(1)}, \ldots, x_{\sigma(N)}) = \text{sign}(\sigma)f(x_1, \ldots, x_N)$, where $\sigma$ is any permutation.
On Non-Linear operators for Geometric Deep Learning
In the context of Neural Networks defined over $\mathcal{M}$, it indicates that point-wise non-linear operators are the only universal family that commutes with any group of symmetries, and justifies their systematic use in combination with dedicated linear operators commuting with specific symmetries.
Beyond the Edge of Stability via Two-step Gradient Updates
Gradient Descent (GD) is a powerful workhorse of modern machine learning thanks to its scalability and efficiency in high-dimensional spaces.
Exponential Separations in Symmetric Neural Networks
In this work we demonstrate a novel separation between symmetric neural network architectures.
When does return-conditioned supervised learning work for offline reinforcement learning?
Several recent works have proposed a class of algorithms for the offline reinforcement learning (RL) problem that we will refer to as return-conditioned supervised learning (RCSL).
On Feature Learning in Neural Networks with Global Convergence Guarantees
We study the optimization of wide neural networks (NNs) via gradient flow (GF) in setups that allow feature learning while admitting non-asymptotic global convergence guarantees.
Neural Galerkin Schemes with Active Learning for High-Dimensional Evolution Equations
Neural Galerkin schemes build on the Dirac-Frenkel variational principle to train networks by minimizing the residual sequentially over time, which enables adaptively collecting new training data in a self-informed manner that is guided by the dynamics described by the partial differential equations.
Extended Unconstrained Features Model for Exploring Deep Neural Collapse
Specifically, it has been shown that the learned features (the output of the penultimate layer) of within-class samples converge to their mean, and the means of different classes exhibit a certain tight frame structure, which is also aligned with the last layer's weights.
Simultaneous Transport Evolution for Minimax Equilibria on Measures
Min-max optimization problems arise in several key machine learning setups, including adversarial learning and generative modeling.
Lattice-Based Methods Surpass Sum-of-Squares in Clustering
Prior work on many similar inference tasks portends that such lower bounds strongly suggest the presence of an inherent statistical-to-computational gap for clustering, that is, a parameter regime where the clustering task is statistically possible but no polynomial-time algorithm succeeds.
Quantile Filtered Imitation Learning
We introduce quantile filtered imitation learning (QFIL), a novel policy improvement operator designed for offline reinforcement learning.
On the Sample Complexity of Learning under Geometric Stability
Many supervised learning problems involve high-dimensional data such as images, text, or graphs.
Neural Fields as Learnable Kernels for 3D Reconstruction
We present Neural Kernel Fields: a novel method for reconstructing implicit 3D shapes based on a learned kernel ridge regression.
Multi-fidelity Stability for Graph Representation Learning
In this example, we provide non-asymptotic bounds that highly depend on the sparsity of the receptive field constructed by the algorithm.
A Rate-Distortion Framework for Explaining Black-box Model Decisions
We present the Rate-Distortion Explanation (RDE) framework, a mathematically well-founded method for explaining black-box model decisions.
Cartoon Explanations of Image Classifiers
We present CartoonX (Cartoon Explanation), a novel model-agnostic explanation method tailored towards image classifiers and based on the rate-distortion explanation (RDE) framework.
On feature learning in shallow and multi-layer neural networks with global convergence guarantees
We study the optimization of over-parameterized shallow and multi-layer neural networks (NNs) in a regime that allows feature learning while admitting non-asymptotic global convergence guarantees.
An Extensible Benchmark Suite for Learning to Simulate Physical Systems
Simulating physical systems is a core component of scientific computing, encompassing a wide range of physical domains and applications.
Dual Training of Energy-Based Models with Overparametrized Shallow Neural Networks
In the feature-learning regime, this dual formulation justifies using a two time-scale gradient ascent-descent (GDA) training algorithm in which one updates concurrently the particles in the sample space and the neurons in the parameter space of the energy.
On the Cryptographic Hardness of Learning Single Periodic Neurons
More precisely, our reduction shows that any polynomial-time algorithm (not necessarily gradient-based) for learning such functions under small noise implies a polynomial-time quantum algorithm for solving worst-case lattice problems, whose hardness form the foundation of lattice-based cryptography.
Offline RL Without Off-Policy Evaluation
In addition, we hypothesize that the strong performance of the one-step algorithm is due to a combination of favorable structure in the environment and behavior policy.
On the Sample Complexity of Learning under Invariance and Geometric Stability
Many supervised learning problems involve high-dimensional data such as images, text, or graphs.
Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges
The last decade has witnessed an experimental revolution in data science and machine learning, epitomised by deep learning methods.
On Energy-Based Models with Overparametrized Shallow Neural Networks
Energy-based models (EBMs) are a simple yet powerful framework for generative modeling.
Symmetry Breaking in Symmetric Tensor Decomposition
In this note, we consider the highly nonconvex optimization problem associated with computing the rank decomposition of symmetric tensors.
Depth separation beyond radial functions
The first contribution of this paper is to extend such results to a more general class of functions, namely functions with piece-wise oscillatory structure, by building on the proof strategy of (Eldan and Shamir, 2016).
Self-Supervised Equivariant Scene Synthesis from Video
We propose a self-supervised framework to learn scene representations from video that are automatically delineated into background, characters, and their animations.
Learned Equivariant Rendering without Transformation Supervision
We propose a self-supervised framework to learn scene representations from video that are automatically delineated into objects and background.
On Graph Neural Networks versus Graph-Augmented MLPs
From the perspective of expressive power, this work compares multi-layer Graph Neural Networks (GNNs) with a simplified alternative that we call Graph-Augmented Multi-Layer Perceptrons (GA-MLPs), which first augments node features with certain multi-hop operators on the graph and then applies an MLP in a node-wise fashion.
Kernel-Based Smoothness Analysis of Residual Networks
A major factor in the success of deep neural networks is the use of sophisticated architectures rather than the classical multilayer perceptron (MLP).
A Dynamical Central Limit Theorem for Shallow Neural Networks
Furthermore, if the mean-field dynamics converges to a measure that interpolates the training data, we prove that the asymptotic deviation eventually vanishes in the CLT scaling.
A Functional Perspective on Learning Symmetric Functions with Neural Networks
Symmetric functions, which take as input an unordered, fixed-size set, are known to be universally representable by neural networks that enforce permutation invariance.
Depth separation for reduced deep networks in nonlinear model reduction: Distilling shock waves in nonlinear hyperbolic problems
Classical reduced models are low-rank approximations using a fixed basis designed to achieve dimensionality reduction of large-scale systems.
In-Distribution Interpretability for Challenging Modalities
It is widely recognized that the predictions of deep neural networks are difficult to parse relative to simpler approaches.
Offline Contextual Bandits with Overparameterized Models
We show that this discrepancy is due to the \emph{action-stability} of their objectives.
Neural Splines: Fitting 3D Surfaces with Infinitely-Wide Neural Networks
We present Neural Splines, a technique for 3D surface reconstruction that is based on random feature kernels arising from infinitely-wide shallow ReLU networks.
On Sparsity in Overparametrised Shallow ReLU Networks
The analysis of neural network training beyond their linearization regime remains an outstanding open question, even in the simplest setup of a single hidden-layer.
IDEAL: Inexact DEcentralized Accelerated Augmented Lagrangian Method
We introduce a framework for designing primal methods under the decentralized optimization setting where local functions are smooth and strongly convex.
Continuous LWE
We introduce a continuous analogue of the Learning with Errors (LWE) problem, which we name CLWE.
A Permutation-Equivariant Neural Network Architecture For Auction Design
Designing an incentive compatible auction that maximizes expected revenue is a central problem in Auction Design.
Provably Efficient Third-Person Imitation from Offline Observation
Domain adaptation in imitation learning represents an essential step towards improving generalizability.
A mean-field analysis of two-player zero-sum games
Our method identifies mixed equilibria in high dimensions and is demonstrably effective for training mixtures of GANs.
Can Graph Neural Networks Count Substructures?
We also prove positive results for k-WL and k-IGNs as well as negative results for k-WL with a finite number of iterations.
Probing the State of the Art: A Critical Look at Visual Representation Evaluation
Our first contribution is to show that this test is insufficient and that models which perform poorly (strongly) on linear classification can perform strongly (weakly) on more involved tasks like temporal activity localization.
Stability of Graph Neural Networks to Relative Perturbations
In this paper, we are set to study the effect that a change in the underlying graph topology that supports the signal has on the output of a GNN.
Pure and Spurious Critical Points: a Geometric Study of Linear Networks
The critical locus of the loss function of a neural network is determined by the geometry of the functional space and by the parameterization of this space by the network's weights.
Projected Canonical Decomposition for Knowledge Base Completion
However, as we show in this paper through experiments on standard benchmarks of link prediction in knowledge bases, ComplEx, a variant of CP, achieves similar performances to recent approaches based on Tucker decomposition on all operating points in terms of number of parameters.
Gradient Dynamics of Shallow Univariate ReLU Networks
We show that the gradient dynamics of such networks are determined by the gradient flow in a non-redundant parameterization of the network function.
Finding the Needle in the Haystack with Convolutions: on the benefits of architectural bias
The aim of this work is to understand this fact through the lens of dynamics in the loss landscape.
Stability of Graph Scattering Transforms
In this work, we extend scattering transforms to network data by using multiresolution graph wavelets, whose computation can be obtained by means of graph convolutions.
Geometric Insights into the Convergence of Nonlinear TD Learning
Then, we show how environments that are more reversible induce dynamics that are better for TD learning and prove global convergence to the true value function for well-conditioned function approximators.
On the Expressive Power of Deep Polynomial Neural Networks
We study deep neural networks with polynomial activations, particularly their expressive power.
On the equivalence between graph isomorphism testing and function approximation with GNNs
We further develop a framework of the expressive power of GNNs that incorporates both of these viewpoints using the language of sigma-algebra, through which we compare the expressive power of different types of GNNs together with other graph isomorphism tests.
Ranked #27 on
Graph Regression
on ZINC-500k
Extragradient with player sampling for faster Nash equilibrium finding
Data-driven modeling increasingly requires to find a Nash equilibrium in multi-player games, e. g. when training GANs.
Stability Properties of Graph Neural Networks
Graph neural networks (GNNs) have emerged as a powerful tool for nonlinear processing of graph signals, exhibiting success in recommender systems, power outage prediction, and motion planning, among others.
Backplay: 'Man muss immer umkehren'
Our contributions are that we analytically characterize the types of environments where Backplay can improve training speed, demonstrate the effectiveness of Backplay both in large grid worlds and a complex four player zero-sum game (Pommerman), and show that Backplay compares favorably to other competitive methods known to improve sample efficiency.
Advancing GraphSAGE with A Data-Driven Node Sampling
As an efficient and scalable graph neural network, GraphSAGE has enabled an inductive capability for inferring unseen nodes or graphs by aggregating subsampled local neighborhoods and by learning in a mini-batch gradient descent fashion.
Global convergence of neuron birth-death dynamics
Neural networks with a large number of parameters admit a mean-field description, which has recently served as a theoretical explanation for the favorable training properties of "overparameterized" models.
Kymatio: Scattering Transforms in Python
The wavelet scattering transform is an invariant signal representation suitable for many signal processing and machine learning applications.
Deep Geometric Prior for Surface Reconstruction
We propose the use of a deep neural network as a geometric prior for surface reconstruction.
Community Detection with Graph Neural Networks
This graph inference task can be recast as a node-wise graph classification problem, and, as such, computational detection thresholds can be translated in terms of learning within appropriate models.
Graph Neural Networks for IceCube Signal Classification
Tasks involving the analysis of geometric (graph- and manifold-structured) data have recently gained prominence in the machine learning community, giving birth to a rapidly developing field of geometric deep learning.
Planning with Arithmetic and Geometric Attributes
A desirable property of an intelligent agent is its ability to understand its environment to quickly generalize to novel tasks and compose simpler tasks into more complex ones.
Backplay: "Man muss immer umkehren"
Our contributions are that we analytically characterize the types of environments where Backplay can improve training speed, demonstrate the effectiveness of Backplay both in large grid worlds and a complex four player zero-sum game (Pommerman), and show that Backplay compares favorably to other competitive methods known to improve sample efficiency.
Spurious Valleys in Two-layer Neural Network Optimization Landscapes
Focusing on a class of two-layer neural networks defined by smooth (but generally non-linear) activation functions, we identify a notion of intrinsic dimension and show that it provides necessary and sufficient conditions for the absence of spurious valleys.
Multiscale Sparse Microcanonical Models
Asymptotic properties of maximum entropy microcanonical and macrocanonical processes and their convergence to Gibbs measures are reviewed.
Mathematics of Deep Learning
Recently there has been a dramatic increase in the performance of recognition systems due to the introduction of deep architectures for representation learning and classification.
Few-Shot Learning with Graph Neural Networks
We propose to study the problem of few-shot learning with the prism of inference on a partially observed graphical model, constructed from a collection of input images whose label can be either observed or not.
Revised Note on Learning Algorithms for Quadratic Assignment with Graph Neural Networks
Inverse problems correspond to a certain type of optimization problems formulated over appropriate input distributions.
Understanding the Learned Iterative Soft Thresholding Algorithm with matrix factorization
Sparse coding is a core building block in many data analysis and machine learning pipelines.
Surface Networks
We study data-driven representations for three-dimensional triangle meshes, which are one of the prevalent objects used to represent 3D geometry.
Supervised Community Detection with Line Graph Neural Networks
We show that, in a data-driven manner and without access to the underlying generative models, they can match or even surpass the performance of the belief propagation algorithm on binary and multi-class stochastic block models, which is believed to reach the computational threshold.
Ranked #1 on
Community Detection
on Amazon
(Accuracy-NE metric, using extra
training data)
Geometric deep learning: going beyond Euclidean data
In many applications, such geometric data are large and complex (in the case of social networks, on the scale of billions), and are natural targets for machine learning techniques.
Divide and Conquer Networks
Moreover, thanks to the dynamic aspect of our architecture, we can incorporate the computational complexity as a regularization term that can be optimized by backpropagation.
Topology and Geometry of Half-Rectified Network Optimization
Our theoretical work quantifies and formalizes two important \emph{folklore} facts: (i) the landscape of deep linear networks has a radically different topology from that of deep half-rectified ones, and (ii) that the energy landscape in the non-linear case is fundamentally controlled by the interplay between the smoothness of the data distribution and model over-parametrization.
Voice Conversion using Convolutional Neural Networks
The human auditory system is able to distinguish the vocal source of thousands of speakers, yet not much is known about what features the auditory system uses to do this.
Inverse Problems with Invariant Multiscale Statistics
We propose a new approach to linear ill-posed inverse problems.
Computational Engineering, Finance, and Science
Understanding Trainable Sparse Coding via Matrix Factorization
Sparse coding is a core building block in many data analysis and machine learning pipelines.
Super-Resolution with Deep Convolutional Sufficient Statistics
Inverse problems in image and audio, and super-resolution in particular, can be seen as high-dimensional structured prediction problems, where the goal is to characterize the conditional distribution of a high-resolution output given its low-resolution corrupted observation.
Deep Convolutional Networks on Graph-Structured Data
Deep Learning's recent successes have mostly relied on Convolutional Networks, which exploit fundamental statistical properties of images, sounds and video data: the local stationarity and multi-scale compositional structure, that allows expressing long range interactions in terms of shorter, localized interactions.
Unsupervised Feature Learning from Temporal Data
Current state-of-the-art classification and detection algorithms rely on supervised training.
A mathematical motivation for complex-valued convolutional networks
Courtesy of the exact correspondence, the remarkably rich and rigorous body of mathematical analysis for wavelets applies directly to (complex-valued) convnets.
Audio Source Separation with Discriminative Scattering Networks
In this report we describe an ongoing line of research for solving single-channel source separation problems.
Video (language) modeling: a baseline for generative models of natural videos
We propose a strong baseline model for unsupervised feature learning using video data.
Unsupervised Learning of Spatiotemporally Coherent Metrics
Current state-of-the-art classification and detection algorithms rely on supervised training.
Training Convolutional Networks with Noisy Labels
The availability of large labeled datasets has allowed Convolutional Network models to achieve impressive recognition results.
Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
We present techniques for speeding up the test-time evaluation of large convolutional networks, designed for object recognition tasks.
Spectral Networks and Locally Connected Networks on Graphs
Convolutional Neural Networks are extremely efficient architectures in image and audio recognition tasks, thanks to their ability to exploit the local translational invariance of signal classes over their domain.
Intriguing properties of neural networks
Deep neural networks are highly expressive models that have recently achieved state of the art performance on speech and visual recognition tasks.
Signal Recovery from Pooling Representations
In this work we compute lower Lipschitz bounds of $\ell_p$ pooling operators for $p=1, 2, \infty$ as well as $\ell_p$ pooling operators preceded by half-rectification layers.
Blind Deconvolution with Non-local Sparsity Reweighting
Blind deconvolution has made significant progress in the past decade.
Invariant Scattering Convolution Networks
A wavelet scattering network computes a translation invariant image representation, which is stable to deformations and preserves high frequency information for classification.
Classification with Scattering Operators
A scattering vector is a local descriptor including multiscale and multi-direction co-occurrence information.
