Search Results for author:
Found 56 papers, 4 papers with code
Private Vector Mean Estimation in the Shuffle Model: Optimal Rates Require Many Messages
We study the problem of private vector mean estimation in the shuffle model of privacy where $n$ users each have a unit vector $v^{(i)} \in\mathbb{R}^d$.
Faster Convergence with Multiway Preferences
We next study a $m$-multiway comparison (`battling') feedback, where the learner can get to see the argmin feedback of $m$-subset of queried points and show a convergence rate of $\smash{\widetilde O}(\frac{d}{ \min\{\log m, d\}\epsilon })$.
Federated Learning with Differential Privacy for End-to-End Speech Recognition
($4. 5$, $10^{-9}$)-$\textbf{DP}$) with a 1. 3% (resp.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Mean Estimation with User-level Privacy under Data Heterogeneity
In this work we propose a simple model of heterogeneous user data that allows user data to differ in both distribution and quantity of data, and provide a method for estimating the population-level mean while preserving user-level differential privacy.
Samplable Anonymous Aggregation for Private Federated Data Analysis
We revisit the problem of designing scalable protocols for private statistics and private federated learning when each device holds its private data.
Differentially Private Heavy Hitter Detection using Federated Analytics
In this work, we study practical heuristics to improve the performance of prefix-tree based algorithms for differentially private heavy hitter detection.
Near-Optimal Algorithms for Private Online Optimization in the Realizable Regime
We also develop an adaptive algorithm for the small-loss setting with regret $O(L^\star\log d + \varepsilon^{-1} \log^{1. 5}{d})$ where $L^\star$ is the total loss of the best expert.
Subspace Recovery from Heterogeneous Data with Non-isotropic Noise
Our goal is to recover the linear subspace shared by $\mu_1,\ldots,\mu_n$ using the data points from all users, where every data point from user $i$ is formed by adding an independent mean-zero noise vector to $\mu_i$.
Private Online Prediction from Experts: Separations and Faster Rates
Our lower bounds also show a separation between pure and approximate differential privacy for adaptive adversaries where the latter is necessary to achieve the non-private $O(\sqrt{T})$ regret.
No Free Lunch in "Privacy for Free: How does Dataset Condensation Help Privacy"
New methods designed to preserve data privacy require careful scrutiny.
Stronger Privacy Amplification by Shuffling for Rényi and Approximate Differential Privacy
Our second contribution is a new analysis of privacy amplification by shuffling.
Optimal Algorithms for Mean Estimation under Local Differential Privacy
We show that PrivUnit (Bhowmick et al. 2018) with optimized parameters achieves the optimal variance among a large family of locally private randomizers.
Private Frequency Estimation via Projective Geometry
In many parameter settings used in practice this is a significant improvement over the $ O(n+k^2)$ computation cost that is achieved by the recent PI-RAPPOR algorithm (Feldman and Talwar; 2021).
Individual Privacy Accounting via a Rényi Filter
In this work, we give a method for tighter privacy loss accounting based on the value of a personalized privacy loss estimate for each individual in each analysis.
Private Stochastic Convex Optimization: Optimal Rates in $\ell_1$ Geometry
Stochastic convex optimization over an $\ell_1$-bounded domain is ubiquitous in machine learning applications such as LASSO but remains poorly understood when learning with differential privacy.
Lossless Compression of Efficient Private Local Randomizers
Here we demonstrate a general approach that, under standard cryptographic assumptions, compresses every efficient LDP algorithm with negligible loss in privacy and utility guarantees.
Hiding Among the Clones: A Simple and Nearly Optimal Analysis of Privacy Amplification by Shuffling
As a direct corollary of our analysis we derive a simple and nearly optimal algorithm for frequency estimation in the shuffle model of privacy.
When is Memorization of Irrelevant Training Data Necessary for High-Accuracy Learning?
Our problems are simple and fairly natural variants of the next-symbol prediction and the cluster labeling tasks.
Individual Privacy Accounting via a Renyi Filter
We consider a sequential setting in which a single dataset of individuals is used to perform adaptively-chosen analyses, while ensuring that the differential privacy loss of each participant does not exceed a pre-specified privacy budget.
What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation
First, natural image and data distributions are (informally) known to be long-tailed, that is have a significant fraction of rare and atypical examples.
Stability of Stochastic Gradient Descent on Nonsmooth Convex Losses
Our work is the first to address uniform stability of SGD on {\em nonsmooth} convex losses.
Private Stochastic Convex Optimization: Optimal Rates in Linear Time
We also give a linear-time algorithm achieving the optimal bound on the excess loss for the strongly convex case, as well as a faster algorithm for the non-smooth case.
PAC learning with stable and private predictions
For $\epsilon$-differentially private prediction we give two new algorithms: one using $\tilde O(d/(\alpha^2\epsilon))$ samples and another one using $\tilde O(d^2/(\alpha\epsilon) + d/\alpha^2)$ samples.
Interaction is necessary for distributed learning with privacy or communication constraints
Our main result is an exponential lower bound on the number of samples necessary to solve the standard task of learning a large-margin linear separator in the non-interactive LDP model.
Private Stochastic Convex Optimization with Optimal Rates
A long line of existing work on private convex optimization focuses on the empirical loss and derives asymptotically tight bounds on the excess empirical loss.
Does Learning Require Memorization? A Short Tale about a Long Tail
In our model, data is sampled from a mixture of subpopulations and our results show that memorization is necessary whenever the distribution of subpopulation frequencies is long-tailed.
The advantages of multiple classes for reducing overfitting from test set reuse
We show a new upper bound of $\tilde O(\max\{\sqrt{k\log(n)/(mn)}, k/n\})$ on the worst-case bias that any attack can achieve in a prediction problem with $m$ classes.
High probability generalization bounds for uniformly stable algorithms with nearly optimal rate
Specifically, their bound on the estimation error of any $\gamma$-uniformly stable learning algorithm on $n$ samples and range in $[0, 1]$ is $O(\gamma \sqrt{n \log(1/\delta)} + \sqrt{\log(1/\delta)/n})$ with probability $\geq 1-\delta$.
Generalization Bounds for Uniformly Stable Algorithms
Specifically, for a loss function with range bounded in $[0, 1]$, the generalization error of a $\gamma$-uniformly stable learning algorithm on $n$ samples is known to be within $O((\gamma +1/n) \sqrt{n \log(1/\delta)})$ of the empirical error with probability at least $1-\delta$.
Amplification by Shuffling: From Local to Central Differential Privacy via Anonymity
We study the collection of such statistics in the local differential privacy (LDP) model, and describe an algorithm whose privacy cost is polylogarithmic in the number of changes to a user's value.
Locally Private Learning without Interaction Requires Separation
The only lower bound we are aware of is for PAC learning an artificial class of functions with respect to a uniform distribution (Kasiviswanathan et al. 2011).
Privacy Amplification by Iteration
In addition, we demonstrate that we can achieve guarantees similar to those obtainable using the privacy-amplification-by-sampling technique in several natural settings where that technique cannot be applied.
Privacy-preserving Prediction
We demonstrate that this overhead can be avoided for the well-studied class of thresholds on a line and for a number of standard settings of convex regression.
The Everlasting Database: Statistical Validity at a Fair Price
The problem of handling adaptivity in data analysis, intentional or not, permeates a variety of fields, including test-set overfitting in ML challenges and the accumulation of invalid scientific discoveries.
Calibrating Noise to Variance in Adaptive Data Analysis
We demonstrate that a simple and natural algorithm based on adding noise scaled to the standard deviation of the query provides our notion of stability.
Generalization for Adaptively-chosen Estimators via Stable Median
We present an algorithm that estimates the expectations of $k$ arbitrary adaptively-chosen real-valued estimators using a number of samples that scales as $\sqrt{k}$.
On the Power of Learning from $k$-Wise Queries
Hence it is natural to ask whether algorithms using $k$-wise queries can solve learning problems more efficiently and by how much.
Dealing with Range Anxiety in Mean Estimation via Statistical Queries
We give algorithms for estimating the expectation of a given real-valued function $\phi:X\to {\bf R}$ on a sample drawn randomly from some unknown distribution $D$ over domain $X$, namely ${\bf E}_{{\bf x}\sim D}[\phi({\bf x})]$.
Generalization of ERM in Stochastic Convex Optimization: The Dimension Strikes Back
In stochastic convex optimization the goal is to minimize a convex function $F(x) \doteq {\mathbf E}_{{\mathbf f}\sim D}[{\mathbf f}(x)]$ over a convex set $\cal K \subset {\mathbb R}^d$ where $D$ is some unknown distribution and each $f(\cdot)$ in the support of $D$ is convex over $\cal K$.
A General Characterization of the Statistical Query Complexity
We give applications of our techniques to two open problems in learning theory and to algorithms that are subject to memory and communication constraints.
Statistical Query Algorithms for Mean Vector Estimation and Stochastic Convex Optimization
Stochastic convex optimization, where the objective is the expectation of a random convex function, is an important and widely used method with numerous applications in machine learning, statistics, operations research and other areas.
Generalization in Adaptive Data Analysis and Holdout Reuse
We also formalize and address the general problem of data reuse in adaptive data analysis.
Tight Bounds on Low-degree Spectral Concentration of Submodular and XOS functions
This improves on previous approaches that all showed an upper bound of $O(1/\epsilon^2)$ for submodular and XOS functions.
Preserving Statistical Validity in Adaptive Data Analysis
We show that, surprisingly, there is a way to estimate an exponential in $n$ number of expectations accurately even if the functions are chosen adaptively.
Agnostic Learning of Disjunctions on Symmetric Distributions
This directly gives an agnostic learning algorithm for disjunctions on symmetric distributions that runs in time $n^{O( \log{(1/\epsilon)})}$.
Approximate resilience, monotonicity, and the complexity of agnostic learning
We study the notion of $\mathit{approximate}$ $\mathit{resilience}$ of Boolean functions, where we say that $f$ is $\alpha$-approximately $d$-resilient if $f$ is $\alpha$-close to a $[-1, 1]$-valued $d$-resilient function in $\ell_1$ distance.
Tight Bounds on $\ell_1$ Approximation and Learning of Self-Bounding Functions
Previous techniques considered stronger $\ell_2$ approximation and proved nearly tight bounds of $\Theta(1/\epsilon^{2})$ on the degree and $2^{\Theta(1/\epsilon^2)}$ on the number of variables.
Sample Complexity Bounds on Differentially Private Learning via Communication Complexity
Our second contribution and the main tool is an equivalence between the sample complexity of (pure) differentially private learning of a concept class $C$ (or $SCDP(C)$) and the randomized one-way communication complexity of the evaluation problem for concepts from $C$.
Statistical Active Learning Algorithms
We describe a framework for designing efficient active learning algorithms that are tolerant to random classification noise.
Optimal Bounds on Approximation of Submodular and XOS Functions by Juntas
This is the first algorithm in the PMAC model that over the uniform distribution can achieve a constant approximation factor arbitrarily close to 1 for all submodular functions.
Statistical Active Learning Algorithms for Noise Tolerance and Differential Privacy
These results combined with our generic conversion lead to the first computationally-efficient algorithms for actively learning some of these concept classes in the presence of random classification noise that provide exponential improvement in the dependence on the error $\epsilon$ over their passive counterparts.
Learning Coverage Functions and Private Release of Marginals
As an application of our learning results, we give simple differentially-private algorithms for releasing monotone conjunction counting queries with low average error.
Representation, Approximation and Learning of Submodular Functions Using Low-rank Decision Trees
We show that these structural results can be exploited to give an attribute-efficient PAC learning algorithm for submodular functions running in time $\tilde{O}(n^2) \cdot 2^{O(1/\epsilon^{4})}$.
Learning using Local Membership Queries
We introduce a new model of membership query (MQ) learning, where the learning algorithm is restricted to query points that are \emph{close} to random examples drawn from the underlying distribution.
Learning DNF Expressions from Fourier Spectrum
This property is crucial for learning of DNF expressions over smoothed product distributions, a learning model introduced by Kalai et al. (2009) and inspired by the seminal smoothed analysis model of Spielman and Teng (2001).
A Complete Characterization of Statistical Query Learning with Applications to Evolvability
The preservation of efficiency is achieved using a new boosting technique and allows us to derive a new approach to the design of evolutionary algorithms in Valiant's (2006) model of evolvability.
