Search Results for author:
Found 70 papers, 16 papers with code
Deep Networks Always Grok and Here is Why
Our local complexity measures the density of the so-called 'linear regions' (aka, spline partition regions) that tile the DNN input space, and serves as a utile progress measure for training.
Learning by Reconstruction Produces Uninformative Features For Perception
Despite interpretability of the reconstruction and generation, we identify a misalignment between learning by reconstruction, and learning for perception.
Fast and Exact Enumeration of Deep Networks Partitions Regions
One fruitful formulation of Deep Networks (DNs) enabling their theoretical study and providing practical guidelines to practitioners relies on Piecewise Affine Splines.
GPS-SSL: Guided Positive Sampling to Inject Prior Into Self-Supervised Learning
Any prior knowledge can now be embedded into that metric space independently from the employed DA.
Characterizing Large Language Model Geometry Solves Toxicity Detection and Generation
We obtain in closed form (i) the intrinsic dimension in which the Multi-Head Attention embeddings are constrained to exist and (ii) the partition and per-region affine mappings of the per-layer feedforward networks.
Training Dynamics of Deep Network Linear Regions
First, we present a novel statistic that encompasses the local complexity (LC) of the DN based on the concentration of linear regions inside arbitrary dimensional neighborhoods around data points.
Martian time-series unraveled: A multi-scale nested approach with factorial variational autoencoders
This problem is inherently ill-posed and is further challenged by the variety of timescales exhibited by sources.
A Cookbook of Self-Supervised Learning
Self-supervised learning, dubbed the dark matter of intelligence, is a promising path to advance machine learning.
A surprisingly simple technique to control the pretraining bias for better transfer: Expand or Narrow your representation
Self-Supervised Learning (SSL) models rely on a pretext task to learn representations.
Active Self-Supervised Learning: A Few Low-Cost Relationships Are All You Need
Third, it provides a proper active learning framework yielding low-cost solutions to annotate datasets, arguably bringing the gap between theory and practice of active learning that is based on simple-to-answer-by-non-experts queries of semantic relationships between inputs.
Towards Democratizing Joint-Embedding Self-Supervised Learning
Joint Embedding Self-Supervised Learning (JE-SSL) has seen rapid developments in recent years, due to its promise to effectively leverage large unlabeled data.
FAIR-Ensemble: When Fairness Naturally Emerges From Deep Ensembling
Ensembling multiple Deep Neural Networks (DNNs) is a simple and effective way to improve top-line metrics and to outperform a larger single model.
An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularization
In this paper, we provide an information-theoretic perspective on Variance-Invariance-Covariance Regularization (VICReg) for self-supervised learning.
SplineCam: Exact Visualization and Characterization of Deep Network Geometry and Decision Boundaries
In this paper, we go one step further by developing the first provably exact method for computing the geometry of a DN's mapping - including its decision boundary - over a specified region of the data space.
Unsupervised Learning on a DIET: Datum IndEx as Target Free of Self-Supervision, Reconstruction, Projector Head
Costly, noisy, and over-specialized, labels are to be set aside in favor of unsupervised learning if we hope to learn cheap, reliable, and transferable models.
The SSL Interplay: Augmentations, Inductive Bias, and Generalization
Self-supervised learning (SSL) has emerged as a powerful framework to learn representations from raw data without supervision.
On minimal variations for unsupervised representation learning
Unsupervised representation learning aims at describing raw data efficiently to solve various downstream tasks.
ImageNet-X: Understanding Model Mistakes with Factor of Variation Annotations
Equipped with ImageNet-X, we investigate 2, 200 current recognition models and study the types of mistakes as a function of model's (1) architecture, e. g. transformer vs. convolutional, (2) learning paradigm, e. g. supervised vs. self-supervised, and (3) training procedures, e. g., data augmentation.
POLICE: Provably Optimal Linear Constraint Enforcement for Deep Neural Networks
In this paper we propose the first provable affine constraint enforcement method for DNNs that only requires minimal changes into a given DNN's forward-pass, that is computationally friendly, and that leaves the optimization of the DNN's parameter to be unconstrained, i. e. standard gradient-based method can be employed.
The Hidden Uniform Cluster Prior in Self-Supervised Learning
A successful paradigm in representation learning is to perform self-supervised pretraining using tasks based on mini-batch statistics (e. g., SimCLR, VICReg, SwAV, MSN).
RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank
Joint-Embedding Self Supervised Learning (JE-SSL) has seen a rapid development, with the emergence of many method variations but only few principled guidelines that would help practitioners to successfully deploy them.
Variance Covariance Regularization Enforces Pairwise Independence in Self-Supervised Representations
Self-Supervised Learning (SSL) methods such as VICReg, Barlow Twins or W-MSE avoid collapse of their joint embedding architectures by constraining or regularizing the covariance matrix of their projector's output.
Batch Normalization Explained
A critically important, ubiquitous, and yet poorly understood ingredient in modern deep networks (DNs) is batch normalization (BN), which centers and normalizes the feature maps.
Joint Embedding Self-Supervised Learning in the Kernel Regime
The fundamental goal of self-supervised learning (SSL) is to produce useful representations of data without access to any labels for classifying the data.
What Do We Maximize in Self-Supervised Learning?
In this paper, we examine self-supervised learning methods, particularly VICReg, to provide an information-theoretical understanding of their construction.
Guillotine Regularization: Why removing layers is needed to improve generalization in Self-Supervised Learning
This is a little vexing, as one would hope that the network layer at which invariance is explicitly enforced by the SSL criterion during training (the last projector layer) should be the one to use for best generalization performance downstream.
Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods
Self-Supervised Learning (SSL) surmises that inputs and pairwise positive relationships are enough to learn meaningful representations.
The Effects of Regularization and Data Augmentation are Class Dependent
The optimal amount of DA or weight decay found from cross-validation leads to disastrous model performances on some classes e. g. on Imagenet with a resnet50, the "barn spider" classification test accuracy falls from $68\%$ to $46\%$ only by introducing random crop DA during training.
DeepTensor: Low-Rank Tensor Decomposition with Deep Network Priors
DeepTensor is a computationally efficient framework for low-rank decomposition of matrices and tensors using deep generative networks.
projUNN: efficient method for training deep networks with unitary matrices
In learning with recurrent or very deep feed-forward networks, employing unitary matrices in each layer can be very effective at maintaining long-range stability.
Singular Value Perturbation and Deep Network Optimization
Building on our earlier work connecting deep networks with continuous piecewise-affine splines, we develop an exact local linear representation of a deep network layer for a family of modern deep networks that includes ConvNets at one end of a spectrum and ResNets, DenseNets, and other networks with skip connections at the other.
No More Than 6ft Apart: Robust K-Means via Radius Upper Bounds
We propose to remedy such a scenario by introducing a maximal radius constraint $r$ on the clusters formed by the centroids, i. e., samples from the same cluster should not be more than $2r$ apart in terms of $\ell_2$ distance.
Polarity Sampling: Quality and Diversity Control of Pre-Trained Generative Networks via Singular Values
We present Polarity Sampling, a theoretically justified plug-and-play method for controlling the generation quality and diversity of pre-trained deep generative networks DGNs).
Ranked #1 on
Image Generation
on LSUN Car 512 x 384
NeuroView-RNN: It's About Time
A key interpretability issue with RNNs is that it is not clear how each hidden state per time step contributes to the decision-making process in a quantitative manner.
Spatial Transformer K-Means
This enables (i) the reduction of intrinsic nuisances associated with the data, reducing the complexity of the clustering task and increasing performances and producing state-of-the-art results, (ii) clustering in the input space of the data, leading to a fully interpretable clustering algorithm, and (iii) the benefit of convergence guarantees.
A Data-Augmentation Is Worth A Thousand Samples: Exact Quantification From Analytical Augmented Sample Moments
We show that for a training loss to be stable under DA sampling, the model's saliency map (gradient of the loss with respect to the model's input) must align with the smallest eigenvector of the sample variance under the considered DA augmentation, hinting at a possible explanation on why models tend to shift their focus from edges to textures.
High Fidelity Visualization of What Your Self-Supervised Representation Knows About
Discovering what is learned by neural networks remains a challenge.
Learning in High Dimension Always Amounts to Extrapolation
The notion of interpolation and extrapolation is fundamental in various fields from deep learning to function approximation.
MaGNET: Uniform Sampling from Deep Generative Network Manifolds Without Retraining
Deep Generative Networks (DGNs) are extensively employed in Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), and their variants to approximate the data manifold and distribution.
Ranked #4 on
Image Generation
on FFHQ 1024 x 1024
NeuroView: Explainable Deep Network Decision Making
Each member of the family is derived from a standard DN architecture by vector quantizing the unit output values and feeding them into a global linear classifier.
Fast Jacobian-Vector Product for Deep Networks
Jacobian-vector products (JVPs) form the backbone of many recent developments in Deep Networks (DNs), with applications including faster constrained optimization, regularization with generalization guarantees, and adversarial example sensitivity assessments.
Max-Affine Spline Insights Into Deep Network Pruning
In this paper, we study the importance of pruning in Deep Networks (DNs) and the yin & yang relationship between (1) pruning highly overparametrized DNs that have been trained from random initialization and (2) training small DNs that have been "cleverly" initialized.
Interpretable Image Clustering via Diffeomorphism-Aware K-Means
We design an interpretable clustering algorithm aware of the nonlinear structure of image manifolds.
Sparse Multi-Family Deep Scattering Network
The SMF-DSN enhances the DSN by (i) increasing the diversity of the scattering coefficients and (ii) improves its robustness with respect to non-stationary noise.
Enhanced Recurrent Neural Tangent Kernels for Non-Time-Series Data
Kernels derived from deep neural networks (DNNs) in the infinite-width regime provide not only high performance in a range of machine learning tasks but also new theoretical insights into DNN training dynamics and generalization.
Analytical Probability Distributions and Exact Expectation-Maximization for Deep Generative Networks
Deep Generative Networks (DGNs) with probabilistic modeling of their output and latent space are currently trained via Variational Autoencoders (VAEs).
Wearing a MASK: Compressed Representations of Variable-Length Sequences Using Recurrent Neural Tangent Kernels
High dimensionality poses many challenges to the use of data, from visualization and interpretation, to prediction and storage for historical preservation.
Deep Autoencoders: From Understanding to Generalization Guarantees
Our regularizations leverage recent advances in the group of transformation learning to enable AEs to better approximate the data manifold without explicitly defining the group underlying the manifold.
Ensembles of Generative Adversarial Networks for Disconnected Data
They can be represented in two ways: With an ensemble of networks or with a single network with truncated latent space.
The Recurrent Neural Tangent Kernel
The study of deep neural networks (DNNs) in the infinite-width limit, via the so-called neural tangent kernel (NTK) approach, has provided new insights into the dynamics of learning, generalization, and the impact of initialization.
Analytical Probability Distributions and EM-Learning for Deep Generative Networks
Deep Generative Networks (DGNs) with probabilistic modeling of their output and latent space are currently trained via Variational Autoencoders (VAEs).
Interpretable Super-Resolution via a Learned Time-Series Representation
We develop an interpretable and learnable Wigner-Ville distribution that produces a super-resolved quadratic signal representation for time-series analysis.
SymJAX: symbolic CPU/GPU/TPU programming
SymJAX is a symbolic programming version of JAX simplifying graph input/output/updates and providing additional functionalities for general machine learning and deep learning applications.
Max-Affine Spline Insights into Deep Generative Networks
We also derive the output probability density mapped onto the generated manifold in terms of the latent space density, which enables the computation of key statistics such as its Shannon entropy.
A GOODNESS OF FIT MEASURE FOR GENERATIVE NETWORKS
We define a goodness of fit measure for generative networks which captures how well the network can generate the training data, which is necessary to learn the true data distribution.
Implicit Rugosity Regularization via Data Augmentation
Deep (neural) networks have been applied productively in a wide range of supervised and unsupervised learning tasks.
The Geometry of Deep Networks: Power Diagram Subdivision
The subdivision process constrains the affine maps on the (exponentially many) power diagram regions to greatly reduce their complexity.
A MAX-AFFINE SPLINE PERSPECTIVE OF RECURRENT NEURAL NETWORKS
Second, we show that the affine parameter of an RNN corresponds to an input-specific template, from which we can interpret an RNN as performing a simple template matching (matched filtering) given the input.
From Hard to Soft: Understanding Deep Network Nonlinearities via Vector Quantization and Statistical Inference
We show that, under a GMM, piecewise affine, convex nonlinearities like ReLU, absolute value, and max-pooling can be interpreted as solutions to certain natural "hard" VQ inference problems, while sigmoid, hyperbolic tangent, and softmax can be interpreted as solutions to corresponding "soft" VQ inference problems.
A Spline Theory of Deep Learning
This implies that a DN constructs a set of signal-dependent, class-specific templates against which the signal is compared via a simple inner product; we explore the links to the classical theory of optimal classification via matched filters and the effects of data memorization.
Spline Filters For End-to-End Deep Learning
We propose to tackle the problem of end-to-end learning for raw waveform signals by introducing learnable continuous time-frequency atoms.
Mad Max: Affine Spline Insights into Deep Learning
For instance, conditioned on the input signal, the output of a MASO DN can be written as a simple affine transformation of the input.
Semi-Supervised Learning Enabled by Multiscale Deep Neural Network Inversion
Deep Neural Networks (DNNs) provide state-of-the-art solutions in several difficult machine perceptual tasks.
Overcomplete Frame Thresholding for Acoustic Scene Analysis
In this work, we derive a generic overcomplete frame thresholding scheme based on risk minimization.
Semi-Supervised Learning via New Deep Network Inversion
We exploit a recently derived inversion scheme for arbitrary deep neural networks to develop a new semi-supervised learning framework that applies to a wide range of systems and problems.
Deep Neural Networks
Deep Neural Networks (DNNs) are universal function approximators providing state-of- the-art solutions on wide range of applications.
Multiscale Residual Mixture of PCA: Dynamic Dictionaries for Optimal Basis Learning
The derived approach lead to a formulation of a Deep Oja Network.
Linear Time Complexity Deep Fourier Scattering Network and Extension to Nonlinear Invariants
In this paper we propose a scalable version of a state-of-the-art deterministic time-invariant feature extraction approach based on consecutive changes of basis and nonlinearities, namely, the scattering network.
Neural Decision Trees
NDT is an architecture a la decision tree where each splitting node is an independent multilayer perceptron allowing oblique decision functions or arbritrary nonlinear decision function if more than one layer is used.
Robust Unsupervised Transient Detection With Invariant Representation based on the Scattering Network
We present a sparse and invariant representation with low asymptotic complexity for robust unsupervised transient and onset zone detection in noisy environments.
