Search Results for author:
Found 33 papers, 19 papers with code
Masked Completion via Structured Diffusion with White-Box Transformers
We do this by exploiting a fundamental connection between diffusion, compression, and (masked) completion, deriving a deep transformer-like masked autoencoder architecture, called CRATE-MAE, in which the role of each layer is mathematically fully interpretable: they transform the data distribution to and from a structured representation.
Differentially Private Representation Learning via Image Captioning
In this work, we show that effective DP representation learning can be done via image captioning and scaling up to internet-scale multimodal datasets.
A Study on the Calibration of In-context Learning
Accurate uncertainty quantification is crucial for the safe deployment of machine learning models, and prior research has demonstrated improvements in the calibration of modern language models (LMs).
White-Box Transformers via Sparse Rate Reduction: Compression Is All There Is?
This leads to a family of white-box transformer-like deep network architectures, named CRATE, which are mathematically fully interpretable.
Emergence of Segmentation with Minimalistic White-Box Transformers
Transformer-like models for vision tasks have recently proven effective for a wide range of downstream applications such as segmentation and detection.
Scaff-PD: Communication Efficient Fair and Robust Federated Learning
We present Scaff-PD, a fast and communication-efficient algorithm for distributionally robust federated learning.
ViP: A Differentially Private Foundation Model for Computer Vision
In this work, we propose as a mitigation measure a recipe to train foundation vision models with differential privacy (DP) guarantee.
White-Box Transformers via Sparse Rate Reduction
Particularly, we show that the standard transformer block can be derived from alternating optimization on complementary parts of this objective: the multi-head self-attention operator can be viewed as a gradient descent step to compress the token sets by minimizing their lossy coding rate, and the subsequent multi-layer perceptron can be viewed as attempting to sparsify the representation of the tokens.
Federated Conformal Predictors for Distributed Uncertainty Quantification
Conformal prediction is emerging as a popular paradigm for providing rigorous uncertainty quantification in machine learning since it can be easily applied as a post-processing step to already trained models.
TCT: Convexifying Federated Learning using Bootstrapped Neural Tangent Kernels
Leveraging this observation, we propose a Train-Convexify-Train (TCT) procedure to sidestep this issue: first, learn features using off-the-shelf methods (e. g., FedAvg); then, optimize a convexified problem obtained from the network's empirical neural tangent kernel approximation.
Robust Calibration with Multi-domain Temperature Scaling
Uncertainty quantification is essential for the reliable deployment of machine learning models to high-stakes application domains.
Conditional Supervised Contrastive Learning for Fair Text Classification
Contrastive representation learning has gained much attention due to its superior performance in learning representations from both image and sequential data.
Online Nonsubmodular Minimization with Delayed Costs: From Full Information to Bandit Feedback
Motivated by applications to online learning in sparse estimation and Bayesian optimization, we consider the problem of online unconstrained nonsubmodular minimization with delayed costs in both full information and bandit feedback settings.
What You See is What You Get: Principled Deep Learning via Distributional Generalization
In contrast, we show that Differentially-Private (DP) training provably ensures the high-level WYSIWYG property, which we quantify using a notion of distributional generalization.
Predicting Out-of-Distribution Error with the Projection Norm
Projection Norm first uses model predictions to pseudo-label test samples and then trains a new model on the pseudo-labels.
The Effect of Model Size on Worst-Group Generalization
Overparameterization is shown to result in poor test accuracy on rare subgroups under a variety of settings where subgroup information is known.
Closed-Loop Data Transcription to an LDR via Minimaxing Rate Reduction
In particular, we propose to learn a closed-loop transcription between a multi-class multi-dimensional data distribution and a linear discriminative representation (LDR) in the feature space that consists of multiple independent multi-dimensional linear subspaces.
An Empirical Study of Pre-trained Models on Out-of-distribution Generalization
Concretely, we show that larger models and larger datasets need to be simultaneously leveraged to improve OOD performance.
On the Convergence of Stochastic Extragradient for Bilinear Games using Restarted Iteration Averaging
We study the stochastic bilinear minimax optimization problem, presenting an analysis of the same-sample Stochastic ExtraGradient (SEG) method with constant step size, and presenting variations of the method that yield favorable convergence.
ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reduction
This work attempts to provide a plausible theoretical framework that aims to interpret modern deep (convolutional) networks from the principles of data compression and discriminative representation.
Fast Distributionally Robust Learning with Variance Reduced Min-Max Optimization
Distributionally robust supervised learning (DRSL) is emerging as a key paradigm for building reliable machine learning systems for real-world applications -- reflecting the need for classifiers and predictive models that are robust to the distribution shifts that arise from phenomena such as selection bias or nonstationarity.
Understanding Generalization in Adversarial Training via the Bias-Variance Decomposition
To investigate this gap, we decompose the test risk into its bias and variance components and study their behavior as a function of adversarial training perturbation radii ($\varepsilon$).
Deep Networks from the Principle of Rate Reduction
The layered architectures, linear and nonlinear operators, and even parameters of the network are all explicitly constructed layer-by-layer in a forward propagation fashion by emulating the gradient scheme.
Adversarial Robustness of Stabilized NeuralODEs Might be from Obfuscated Gradients
Even replacing only the first layer of a ResNet by such a ODE block can exhibit further improvement in robustness, e. g., under PGD-20 ($\ell_\infty=0. 031$) attack on CIFAR-10 dataset, it achieves 91. 57\% and natural accuracy and 62. 35\% robust accuracy, while a counterpart architecture of ResNet trained with TRADES achieves natural and robust accuracy 76. 29\% and 45. 24\%, respectively.
Boundary thickness and robustness in learning models
Using these observations, we show that noise-augmentation on mixup training further increases boundary thickness, thereby combating vulnerability to various forms of adversarial attacks and OOD transforms.
Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction
To learn intrinsic low-dimensional structures from high-dimensional data that most discriminate between classes, we propose the principle of Maximal Coding Rate Reduction ($\text{MCR}^2$), an information-theoretic measure that maximizes the coding rate difference between the whole dataset and the sum of each individual class.
Ranked #15 on
Image Clustering
on STL-10
Rethinking Bias-Variance Trade-off for Generalization of Neural Networks
We provide a simple explanation for this by measuring the bias and variance of neural networks: while the bias is monotonically decreasing as in the classical theory, the variance is unimodal or bell-shaped: it increases then decreases with the width of the network.
Theoretically Principled Trade-off between Robustness and Accuracy
We identify a trade-off between robustness and accuracy that serves as a guiding principle in the design of defenses against adversarial examples.
Ranked #3 on
Adversarial Attack
on CIFAR-10
Third-order Smoothness Helps: Faster Stochastic Optimization Algorithms for Finding Local Minima
We propose stochastic optimization algorithms that can find local minima faster than existing algorithms for nonconvex optimization problems, by exploiting the third-order smoothness to escape non-degenerate saddle points more efficiently.
A Primal-Dual Analysis of Global Optimality in Nonconvex Low-Rank Matrix Recovery
We propose a primal-dual based framework for analyzing the global optimality of nonconvex low-rank matrix recovery.
Learning One-hidden-layer ReLU Networks via Gradient Descent
We study the problem of learning one-hidden-layer neural networks with Rectified Linear Unit (ReLU) activation function, where the inputs are sampled from standard Gaussian distribution and the outputs are generated from a noisy teacher network.
Third-order Smoothness Helps: Even Faster Stochastic Optimization Algorithms for Finding Local Minima
We propose stochastic optimization algorithms that can find local minima faster than existing algorithms for nonconvex optimization problems, by exploiting the third-order smoothness to escape non-degenerate saddle points more efficiently.
Saving Gradient and Negative Curvature Computations: Finding Local Minima More Efficiently
We propose a family of nonconvex optimization algorithms that are able to save gradient and negative curvature computations to a large extent, and are guaranteed to find an approximate local minimum with improved runtime complexity.
