Search Results for author:
Found 20 papers, 8 papers with code
Max-Margin Works while Large Margin Fails: Generalization without Uniform Convergence
Nagarajan and Kolter (2019) show that in certain simple linear and neural-network settings, any uniform convergence bound will be vacuous, leaving open the question of how to prove generalization in settings where UC fails.
Beyond Separability: Analyzing the Linear Transferability of Contrastive Representations to Related Subpopulations
In particular, a linear classifier trained to separate the representations on the source domain can also predict classes on the target domain accurately, even though the representations of the two domains are far from each other.
Statistically Meaningful Approximation: a Theoretical Analysis for Approximating Turing Machines with Transformers
A common lens to theoretically study neural net architectures is to analyze the functions they can approximate.
Certified Robustness for Deep Equilibrium Models via Interval Bound Propagation
Our key insights are that these interval bounds can be obtained as the fixed-point solution to an IBP-inspired equilibrium equation, and furthermore, that this solution always exists and is unique when the layer obeys a certain parameterization.
Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformers
A common lens to theoretically study neural net architectures is to analyze the functions they can approximate.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
The generative model in our analysis is either a Hidden Markov Model (HMM) or an HMM augmented with a latent memory component, motivated by long-term dependencies in natural language.
Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss
Despite the empirical successes, theoretical foundations are limited -- prior analyses assume conditional independence of the positive pairs given the same class label, but recent empirical applications use heavily correlated positive pairs (i. e., data augmentations of the same image).
Meta-learning Transferable Representations with a Single Target Domain
Recent works found that fine-tuning and joint training---two popular approaches for transfer learning---do not always improve accuracy on downstream tasks.
Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data
Self-training algorithms, which train a model to fit pseudolabels predicted by another previously-learned model, have been very successful for learning with unlabeled data using neural networks.
Self-training Avoids Using Spurious Features Under Domain Shift
In unsupervised domain adaptation, existing theory focuses on situations where the source and target domains are close.
Shape Matters: Understanding the Implicit Bias of the Noise Covariance
We show that in an over-parameterized setting, SGD with label noise recovers the sparse ground-truth with an arbitrary initialization, whereas SGD with Gaussian noise or gradient descent overfits to dense solutions with large norms.
Improved Sample Complexities for Deep Neural Networks and Robust Classification via an All-Layer Margin
For linear classifiers, the relationship between (normalized) output margin and generalization is captured in a clear and simple bound – a large output margin implies good generalization.
The Implicit and Explicit Regularization Effects of Dropout
This implicit regularization effect is analogous to the effect of stochasticity in small mini-batch stochastic gradient descent.
Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margin
Unfortunately, for deep models, this relationship is less clear: existing analyses of the output margin give complicated bounds which sometimes depend exponentially on depth.
Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks
This concept translates to a larger-scale setting: we demonstrate that one can add a small patch to CIFAR-10 images that is immediately memorizable by a model with small initial learning rate, but ignored by the model with large learning rate until after annealing.
Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss
Deep learning algorithms can fare poorly when the training dataset suffers from heavy class-imbalance but the testing criterion requires good generalization on less frequent classes.
Ranked #4 on
Long-tail learning with class descriptors
on CUB-LT
Data-dependent Sample Complexity of Deep Neural Networks via Lipschitz Augmentation
For feedforward neural nets as well as RNNs, we obtain tighter Rademacher complexity bounds by considering additional data-dependent properties of the network: the norms of the hidden layers of the network, and the norms of the Jacobians of each layer with respect to all previous layers.
On the Margin Theory of Feedforward Neural Networks
We establish: 1) for multi-layer feedforward relu networks, the global minimizer of a weakly-regularized cross-entropy loss has the maximum normalized margin among all networks, 2) as a result, increasing the over-parametrization improves the normalized margin and generalization error bounds for deep networks.
Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel
We prove that for infinite-width two-layer nets, noisy gradient descent optimizes the regularized neural net loss to a global minimum in polynomial iterations.
Markov Chain Truncation for Doubly-Intractable Inference
Computing partition functions, the normalizing constants of probability distributions, is often hard.
