Search Results for author:
Found 25 papers, 4 papers with code
Simplicity Bias of Transformers to Learn Low Sensitivity Functions
Transformers achieve state-of-the-art accuracy and robustness across many tasks, but an understanding of the inductive biases that they have and how those biases are different from other neural network architectures remains elusive.
Learnability is a Compact Property
Furthermore, the learnability of such problems can fail to be a property of finite character: informally, it cannot be detected by examining finite projections of the problem.
Stability and Multigroup Fairness in Ranking with Uncertain Predictions
However, when predictors trained for classification tasks have intrinsic uncertainty, it is not obvious how this uncertainty should be represented in the derived rankings.
Transformers Learn Higher-Order Optimization Methods for In-Context Learning: A Study with Linear Models
In this paper, we instead demonstrate that Transformers learn to implement higher-order optimization methods to perform ICL.
Mitigating Simplicity Bias in Deep Learning for Improved OOD Generalization and Robustness
Neural networks (NNs) are known to exhibit simplicity bias where they tend to prefer learning 'simple' features over more 'complex' ones, even when the latter may be more informative.
Regularization and Optimal Multiclass Learning
We demonstrate that an agnostic version of the Hall complexity again characterizes error rates exactly, and exhibit an optimal learner using maximum entropy programs.
Fairness in Matching under Uncertainty
The prevalence and importance of algorithmic two-sided marketplaces has drawn attention to the issue of fairness in such settings.
Efficient Convex Optimization Requires Superlinear Memory
We show that any memory-constrained, first-order algorithm which minimizes $d$-dimensional, $1$-Lipschitz convex functions over the unit ball to $1/\mathrm{poly}(d)$ accuracy using at most $d^{1. 25 - \delta}$ bits of memory must make at least $\tilde{\Omega}(d^{1 + (4/3)\delta})$ first-order queries (for any constant $\delta \in [0, 1/4]$).
KL Divergence Estimation with Multi-group Attribution
Estimating the Kullback-Leibler (KL) divergence between two distributions given samples from them is well-studied in machine learning and information theory.
On the Statistical Complexity of Sample Amplification
In this work, we place the sample amplification problem on a firm statistical foundation by deriving generally applicable amplification procedures, lower bound techniques and connections to existing statistical notions.
Big-Step-Little-Step: Efficient Gradient Methods for Objectives with Multiple Scales
We consider the problem of minimizing a function $f : \mathbb{R}^d \rightarrow \mathbb{R}$ which is implicitly decomposable as the sum of $m$ unknown non-interacting smooth, strongly convex functions and provide a method which solves this problem with a number of gradient evaluations that scales (up to logarithmic factors) as the product of the square-root of the condition numbers of the components.
Omnipredictors
We suggest a rigorous new paradigm for loss minimization in machine learning where the loss function can be ignored at the time of learning and only be taken into account when deciding an action.
One Network Fits All? Modular versus Monolithic Task Formulations in Neural Networks
Can deep learning solve multiple tasks simultaneously, even when they are unrelated and very different?
Multicalibrated Partitions for Importance Weights
We significantly strengthen previous work that use the MaxEntropy approach, that define the importance weights based on a distribution $Q$ closest to $P$, that looks the same as $R$ on every set $C \in \mathcal{C}$, where $\mathcal{C}$ may be a huge collection of sets.
PIDForest: Anomaly Detection via Partial Identification
We consider the problem of detecting anomalies in a large dataset.
Sample Amplification: Increasing Dataset Size even when Learning is Impossible
In the Gaussian case, we show that an $\left(n, n+\Theta(\frac{n}{\sqrt{d}} )\right)$ amplifier exists, even though learning the distribution to small constant total variation distance requires $\Theta(d)$ samples.
Memory-Sample Tradeoffs for Linear Regression with Small Error
We consider the problem of performing linear regression over a stream of $d$-dimensional examples, and show that any algorithm that uses a subquadratic amount of memory exhibits a slower rate of convergence than can be achieved without memory constraints.
A Spectral View of Adversarially Robust Features
This connection can be leveraged to provide both robust features, and a lower bound on the robustness of any function that has significant variance across the dataset.
Recovery Guarantees for Quadratic Tensors with Sparse Observations
We consider the tensor completion problem of predicting the missing entries of a tensor.
Efficient Anomaly Detection via Matrix Sketching
We consider the problem of finding anomalies in high-dimensional data using popular PCA based anomaly scores.
Sketching Linear Classifiers over Data Streams
We introduce a new sub-linear space sketch---the Weight-Median Sketch---for learning compressed linear classifiers over data streams while supporting the efficient recovery of large-magnitude weights in the model.
Learning Overcomplete HMMs
On the other hand, we show that learning is impossible given only a polynomial number of samples for HMMs with a small output alphabet and whose transition matrices are random regular graphs with large degree.
Compressed Factorization: Fast and Accurate Low-Rank Factorization of Compressively-Sensed Data
What learning algorithms can be run directly on compressively-sensed data?
Orthogonalized ALS: A Theoretically Principled Tensor Decomposition Algorithm for Practical Use
The popular Alternating Least Squares (ALS) algorithm for tensor decomposition is efficient and easy to implement, but often converges to poor local optima---particularly when the weights of the factors are non-uniform.
Prediction with a Short Memory
For a Hidden Markov Model with $n$ hidden states, $I$ is bounded by $\log n$, a quantity that does not depend on the mixing time, and we show that the trivial prediction algorithm based on the empirical frequencies of length $O(\log n/\epsilon)$ windows of observations achieves this error, provided the length of the sequence is $d^{\Omega(\log n/\epsilon)}$, where $d$ is the size of the observation alphabet.
