Search Results for author:
Found 52 papers, 11 papers with code
Toward a Theory of Tokenization in LLMs
In this paper, we investigate tokenization from a theoretical point of view by studying the behavior of transformers on simple data generating processes.
Generative AI Security: Challenges and Countermeasures
Generative AI's expanding footprint across numerous industries has led to both excitement and increased scrutiny.
Efficient Prompt Caching via Embedding Similarity
In this paper, we aim to improve the inference efficiency of LLMs by prompt caching, i. e., if the current prompt can be answered by the same response of a previous prompt, one can directly utilize that previous response without calling the LLM.
Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF
Reinforcement Learning from Human Feedback (RLHF) is a pivotal technique that aligns language models closely with human-centric values.
Towards Optimal Statistical Watermarking
Key to our formulation is a coupling of the output tokens and the rejection region, realized by pseudo-random generators in practice, that allows non-trivial trade-offs between the Type I error and Type II error.
End-to-end Story Plot Generator
Story plots, while short, carry most of the essential information of a full story that may contain tens of thousands of words.
Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignment
In summary, this work introduces a simpler yet effective approach for aligning LLMs to human preferences through relative feedback.
Guided Online Distillation: Promoting Safe Reinforcement Learning by Offline Demonstration
GOLD distills an offline DT policy into a lightweight policy network through guided online safe RL training, which outperforms both the offline DT policy and online safe RL algorithms.
Noisy Computing of the $\mathsf{OR}$ and $\mathsf{MAX}$ Functions
We consider the problem of computing a function of $n$ variables using noisy queries, where each query is incorrect with some fixed and known probability $p \in (0, 1/2)$.
On the Optimal Bounds for Noisy Computing
However, the upper and lower bounds do not match in terms of the dependence on $\delta$ and $p$.
Fine-Tuning Language Models with Advantage-Induced Policy Alignment
Reinforcement learning from human feedback (RLHF) has emerged as a reliable approach to aligning large language models (LLMs) to human preferences.
On Optimal Caching and Model Multiplexing for Large Model Inference
Theoretically, we provide an optimal algorithm for jointly optimizing both approaches to reduce the inference cost in both offline and online tabular settings.
Doubly Robust Self-Training
Self-training is an important technique for solving semi-supervised learning problems.
Online Learning in a Creator Economy
In this paper, we initiate the study of online learning in the creator economy by modeling the creator economy as a three-party game between the users, platform, and content creators, with the platform interacting with the content creator under a principal-agent model through contracts to encourage better content.
Statistical Complexity and Optimal Algorithms for Non-linear Ridge Bandits
We consider the sequential decision-making problem where the mean outcome is a non-linear function of the chosen action.
Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learning
We propose A-Crab (Actor-Critic Regularized by Average Bellman error), a new practical algorithm for offline reinforcement learning (RL) in complex environments with insufficient data coverage.
Online Learning in Stackelberg Games with an Omniscient Follower
We analyze the sample complexity of regret minimization in this repeated Stackelberg game.
Principled Reinforcement Learning with Human Feedback from Pairwise or $K$-wise Comparisons
Our analysis shows that when the true reward function is linear, the widely used maximum likelihood estimator (MLE) converges under both the Bradley-Terry-Luce (BTL) model and the Plackett-Luce (PL) model.
The Sample Complexity of Online Contract Design
This result shows that exponential-in-$m$ samples are sufficient and necessary to learn a near-optimal contract, resolving an open problem on the hardness of online contract design.
Beyond the Best: Estimating Distribution Functionals in Infinite-Armed Bandits
In the infinite-armed bandit problem, each arm's average reward is sampled from an unknown distribution, and each arm can be sampled further to obtain noisy estimates of the average reward of that arm.
Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian
Offline reinforcement learning (RL), which refers to decision-making from a previously-collected dataset of interactions, has received significant attention over the past years.
Minimax Optimal Online Imitation Learning via Replay Estimation
In the tabular setting or with linear function approximation, our meta theorem shows that the performance gap incurred by our approach achieves the optimal $\widetilde{O} \left( \min({H^{3/2}} / {N}, {H} / {\sqrt{N}} \right)$ dependency, under significantly weaker assumptions compared to prior work.
Byzantine-Robust Federated Learning with Optimal Statistical Rates and Privacy Guarantees
In contrast to prior work, our proposed protocols improve the dimension dependence and achieve a tight statistical rate in terms of all the parameters for strongly convex losses.
Jump-Start Reinforcement Learning
In addition, we provide an upper bound on the sample complexity of JSRL and show that with the help of a guide-policy, one can improve the sample complexity for non-optimism exploration methods from exponential in horizon to polynomial.
Robust Estimation for Nonparametric Families via Generative Adversarial Networks
Prior work focus on the problem of robust mean and covariance estimation when the true distribution lies in the family of Gaussian distributions or elliptical distributions, and analyze depth or scoring rule based GAN losses for the problem.
Nearly Optimal Policy Optimization with Stable at Any Time Guarantee
Policy optimization methods are one of the most widely used classes of Reinforcement Learning (RL) algorithms.
On the Value of Interaction and Function Approximation in Imitation Learning
In contrast, when the MDP transition structure is known to the learner such as in the case of simulators, we demonstrate fundamental differences compared to the tabular setting in terms of the performance of an optimal algorithm, Mimic-MD (Rajaraman et al. (2020)) when extended to the function approximation setting.
Computational Benefits of Intermediate Rewards for Goal-Reaching Policy Learning
In both OWSP and OWMP settings, we demonstrate that adding {\em intermediate rewards} to subgoals is more computationally efficient than only rewarding the agent once it completes the goal of reaching a terminal state.
MADE: Exploration via Maximizing Deviation from Explored Regions
As a proof of concept, we evaluate the new intrinsic reward on tabular examples across a variety of model-based and model-free algorithms, showing improvements over count-only exploration strategies.
Securing Secure Aggregation: Mitigating Multi-Round Privacy Leakage in Federated Learning
In fact, we show that the conventional random user selection strategies in FL lead to leaking users' individual models within number of rounds that is linear in the number of users.
Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism
Based on the composition of the offline dataset, two main categories of methods are used: imitation learning which is suitable for expert datasets and vanilla offline RL which often requires uniform coverage datasets.
Provably Breaking the Quadratic Error Compounding Barrier in Imitation Learning, Optimally
We establish an upper bound $O(|\mathcal{S}|H^{3/2}/N)$ for the suboptimality using the Mimic-MD algorithm in Rajaraman et al (2020) which we prove to be computationally efficient.
Minimax Off-Policy Evaluation for Multi-Armed Bandits
Second, when the behavior policy is unknown, we analyze performance in terms of the competitive ratio, thereby revealing a fundamental gap between the settings of known and unknown behavior policies.
Linear Representation Meta-Reinforcement Learning for Instant Adaptation
This paper introduces Fast Linearized Adaptive Policy (FLAP), a new meta-reinforcement learning (meta-RL) method that is able to extrapolate well to out-of-distribution tasks without the need to reuse data from training, and adapt almost instantaneously with the need of only a few samples during testing.
SLIP: Learning to Predict in Unknown Dynamical Systems with Long-Term Memory
Our theoretical and experimental results shed light on the conditions required for efficient probably approximately correct (PAC) learning of the Kalman filter from partially observed data.
Toward the Fundamental Limits of Imitation Learning
Here, we show that the policy which mimics the expert whenever possible is in expectation $\lesssim \frac{|\mathcal{S}| H^2 \log (N)}{N}$ suboptimal compared to the value of the expert, even when the expert follows an arbitrary stochastic policy.
Robust estimation via generalized quasi-gradients
We study the loss landscape of these robust estimation problems, and identify the existence of "generalized quasi-gradients".
When does the Tukey median work?
We show that under TV corruptions, the breakdown point reduces to 1/4 for the same set of distributions.
Generalized Resilience and Robust Statistics
This generalizes a property called resilience previously employed in the special case of mean estimation with outliers.
Deconstructing Generative Adversarial Networks
Generalization: given a population target of GANs, we design a systematic principle, projection under admissible distance, to design GANs to meet the population requirement using finite samples.
Theoretically Principled Trade-off between Robustness and Accuracy
We identify a trade-off between robustness and accuracy that serves as a guiding principle in the design of defenses against adversarial examples.
Ranked #3 on
Adversarial Attack
on CIFAR-10
Stackelberg GAN: Towards Provable Minimax Equilibrium via Multi-Generator Architectures
In this work, we give new results on the benefits of multi-generator architecture of GANs.
Local moment matching: A unified methodology for symmetric functional estimation and distribution estimation under Wasserstein distance
We present \emph{Local Moment Matching (LMM)}, a unified methodology for symmetric functional estimation and distribution estimation under Wasserstein distance.
Entropy Rate Estimation for Markov Chains with Large State Space
For estimating the Shannon entropy of a distribution on $S$ elements with independent samples, [Paninski2004] showed that the sample complexity is sublinear in $S$, and [Valiant--Valiant2011] showed that consistent estimation of Shannon entropy is possible if and only if the sample size $n$ far exceeds $\frac{S}{\log S}$.
Approximate Profile Maximum Likelihood
We propose an efficient algorithm for approximate computation of the profile maximum likelihood (PML), a variant of maximum likelihood maximizing the probability of observing a sufficient statistic rather than the empirical sample.
The Nearest Neighbor Information Estimator is Adaptively Near Minimax Rate-Optimal
We analyze the Kozachenko--Leonenko (KL) nearest neighbor estimator for the differential entropy.
On Estimation of $L_{r}$-Norms in Gaussian White Noise Models
We provide a complete picture of asymptotically minimax estimation of $L_r$-norms (for any $r\ge 1$) of the mean in Gaussian white noise model over Nikolskii-Besov spaces.
Bias Correction with Jackknife, Bootstrap, and Taylor Series
We analyze bias correction methods using jackknife, bootstrap, and Taylor series.
Estimating the Fundamental Limits is Easier than Achieving the Fundamental Limits
We show through case studies that it is easier to estimate the fundamental limits of data processing than to construct explicit algorithms to achieve those limits.
Demystifying ResNet
We show that with or without nonlinearities, by adding shortcuts that have depth two, the condition number of the Hessian of the loss function at the zero initial point is depth-invariant, which makes training very deep models no more difficult than shallow ones.
Beyond Maximum Likelihood: from Theory to Practice
In a nutshell, a message of this recent work is that, for a wide class of functionals, the performance of these essentially optimal estimators with $n$ samples is comparable to that of the MLE with $n \ln n$ samples.
Universal Estimation of Directed Information
Four estimators of the directed information rate between a pair of jointly stationary ergodic finite-alphabet processes are proposed, based on universal probability assignments.
Information Theory Information Theory
