Search Results for author:
Found 99 papers, 12 papers with code
Transformers Provably Learn Feature-Position Correlations in Masked Image Modeling
Masked image modeling (MIM), which predicts randomly masked patches from unmasked ones, has emerged as a promising approach in self-supervised vision pretraining.
Take the Bull by the Horns: Hard Sample-Reweighted Continual Training Improves LLM Generalization
Our study starts from an empirical strategy for the light continual training of LLMs using their original pre-training data sets, with a specific focus on selective retention of samples that incur moderately high losses.
Non-asymptotic Convergence of Discrete-time Diffusion Models: New Approach and Improved Rate
In this paper, we establish the convergence guarantee for substantially larger classes of distributions under discrete-time diffusion models and further improve the convergence rate for distributions with bounded support.
Sample Complexity Characterization for Linear Contextual MDPs
Our result for the second model is the first-known result for such a type of function approximation models.
Rethinking PGD Attack: Is Sign Function Necessary?
Based on that, we further propose a new raw gradient descent (RGD) algorithm that eliminates the use of sign.
Meta ControlNet: Enhancing Task Adaptation via Meta Learning
However, vanilla ControlNet generally requires extensive training of around 5000 steps to achieve a desirable control for a single task.
Provable Benefits of Multi-task RL under Non-Markovian Decision Making Processes
To this end, we posit a joint model class for tasks and use the notion of $\eta$-bracketing number to quantify its complexity; this number also serves as a general metric to capture the similarity of tasks and thus determines the benefit of multi-task over single-task RL.
In-Context Convergence of Transformers
For data with balanced features, we establish the finite-time convergence guarantee with near-zero prediction error by navigating our analysis over two phases of the training dynamics of the attention map.
Model-Free Algorithm with Improved Sample Efficiency for Zero-Sum Markov Games
In this work, we propose a model-free stage-based Q-learning algorithm and show that it achieves the same sample complexity as the best model-based algorithm, and hence for the first time demonstrate that model-free algorithms can enjoy the same optimality in the $H$ dependence as model-based algorithms.
Doubly Robust Instance-Reweighted Adversarial Training
Experiments on standard classification datasets demonstrate that our proposed approach outperforms related state-of-the-art baseline methods in terms of average robust performance, and at the same time improves the robustness against attacks on the weakest data points.
Provably Efficient UCB-type Algorithms For Learning Predictive State Representations
The general sequential decision-making problem, which includes Markov decision processes (MDPs) and partially observable MDPs (POMDPs) as special cases, aims at maximizing a cumulative reward by making a sequence of decisions based on a history of observations and actions over time.
Theoretical Hardness and Tractability of POMDPs in RL with Partial Online State Information
However, existing theoretical results have shown that learning in POMDPs is intractable in the worst case, where the main challenge lies in the lack of latent state information.
Generalization Performance of Transfer Learning: Overparameterized and Underparameterized Regimes
For example, when the total number of features in the source task's learning model is fixed, we show that it is more advantageous to allocate a greater number of redundant features to the task-specific part rather than the common part.
Non-stationary Reinforcement Learning under General Function Approximation
To the best of our knowledge, this is the first dynamic regret analysis in non-stationary MDPs with general function approximation.
Theoretical Characterization of the Generalization Performance of Overfitted Meta-Learning
However, due to the uniqueness of meta-learning such as task-specific gradient descent inner training and the diversity/fluctuation of the ground-truth signals among training tasks, we find new and interesting properties that do not exist in single-task linear regression.
Improved Sample Complexity for Reward-free Reinforcement Learning under Low-rank MDPs
In this work, we first provide the first known sample complexity lower bound that holds for any algorithm under low-rank MDPs.
M-L2O: Towards Generalizable Learning-to-Optimize by Test-Time Fast Self-Adaptation
This data-driven procedure yields L2O that can efficiently solve problems similar to those seen in training, that is, drawn from the same ``task distribution".
Learning to Generalize Provably in Learning to Optimize
While the optimizer generalization has been recently studied, the optimizee generalization (or learning to generalize) has not been rigorously studied in the L2O context, which is the aim of this paper.
Theory on Forgetting and Generalization of Continual Learning
In particular, there is a lack of understanding on what factors are important and how they affect "catastrophic forgetting" and generalization performance.
A Near-Optimal Algorithm for Safe Reinforcement Learning Under Instantaneous Hard Constraints
In many applications of Reinforcement Learning (RL), it is critically important that the algorithm performs safely, such that instantaneous hard constraints are satisfied at each step, and unsafe states and actions are avoided.
Near-Optimal Adversarial Reinforcement Learning with Switching Costs
Our lower bound indicates that, due to the fundamental challenge of switching costs in adversarial RL, the best achieved regret (whose dependency on $T$ is $\tilde{O}(\sqrt{T})$) in static RL with switching costs (as well as adversarial RL without switching costs) is no longer achievable.
Algorithm Design for Online Meta-Learning with Task Boundary Detection
More specifically, we first propose two simple but effective detection mechanisms of task switches and distribution shift based on empirical observations, which serve as a key building block for more elegant online model updates in our algorithm: the task switch detection mechanism allows reusing of the best model available for the current task at hand, and the distribution shift detection mechanism differentiates the meta model update in order to preserve the knowledge for in-distribution tasks and quickly learn the new knowledge for out-of-distribution tasks.
Pruning Before Training May Improve Generalization, Provably
It is shown that as long as the pruning fraction is below a certain threshold, gradient descent can drive the training loss toward zero and the network exhibits good generalization performance.
Convergence and Generalization of Wide Neural Networks with Large Bias
This work studies training one-hidden-layer overparameterized ReLU networks via gradient descent in the neural tangent kernel (NTK) regime, where the networks' biases are initialized to some constant rather than zero.
Global Convergence of Two-timescale Actor-Critic for Solving Linear Quadratic Regulator
To our knowledge, this is the first finite-time convergence analysis for the single sample two-timescale AC for solving LQR with global optimality.
Safe Exploration Incurs Nearly No Additional Sample Complexity for Reward-free RL
In particular, we consider the scenario where a safe baseline policy is known beforehand, and propose a unified Safe reWard-frEe ExploraTion (SWEET) framework.
Provable Generalization of Overparameterized Meta-learning Trained with SGD
Despite the superior empirical success of deep meta-learning, theoretical understanding of overparameterized meta-learning is still limited.
Provable Benefit of Multitask Representation Learning in Reinforcement Learning
To the best of our knowledge, this is the first theoretical study that characterizes the benefit of representation learning in exploration-based reward-free multitask RL for both upstream and downstream tasks.
Provably Efficient Offline Reinforcement Learning with Trajectory-Wise Reward
The remarkable success of reinforcement learning (RL) heavily relies on observing the reward of every visited state-action pair.
Will Bilevel Optimizers Benefit from Loops
Existing studies in the literature cover only some of those implementation choices, and the complexity bounds available are not refined enough to enable rigorous comparison among different implementations.
Data Sampling Affects the Complexity of Online SGD over Dependent Data
Moreover, we show that online SGD with mini-batch sampling can further substantially improve the sample complexity over online SGD with periodic data-subsampling over highly dependent data.
A Primal-Dual Approach to Bilevel Optimization with Multiple Inner Minima
Existing algorithms designed for such a problem were applicable to restricted situations and do not come with a full guarantee of convergence.
Model-Based Offline Meta-Reinforcement Learning with Regularization
In particular, we devise a new meta-Regularized model-based Actor-Critic (RAC) method for within-task policy optimization, as a key building block of MerPO, using conservative policy evaluation and regularized policy improvement; and the intrinsic tradeoff therein is achieved via striking the right balance between two regularizers, one based on the behavior policy and the other on the meta-policy.
Faster Non-asymptotic Convergence for Double Q-learning
This paper tackles the more challenging case of a constant learning rate, and develops new analytical tools that improve the existing convergence rate by orders of magnitude.
Faster Algorithm and Sharper Analysis for Constrained Markov Decision Process
Despite the challenge of the nonconcave objective subject to nonconcave constraints, the proposed approach is shown to converge to the global optimum with a complexity of $\tilde{\mathcal O}(1/\epsilon)$ in terms of the optimality gap and the constraint violation, which improves the complexity of the existing primal-dual approach by a factor of $\mathcal O(1/\epsilon)$ \citep{ding2020natural, paternain2019constrained}.
PER-ETD: A Polynomially Efficient Emphatic Temporal Difference Learning Method
Although ETD has been shown to converge asymptotically to a desirable value function, it is well-known that ETD often encounters a large variance so that its sample complexity can increase exponentially fast with the number of iterations.
On the Convergence Theory for Hessian-Free Bilevel Algorithms
Bilevel optimization has arisen as a powerful tool in modern machine learning.
How to Improve Sample Complexity of SGD over Highly Dependent Data?
Specifically, with a $\phi$-mixing model that captures both exponential and polynomial decay of the data dependence over time, we show that SGD with periodic data-subsampling achieves an improved sample complexity over the standard SGD in the full spectrum of the $\phi$-mixing data dependence.
Generalizable Learning to Optimize into Wide Valleys
Learning to optimize (L2O) has gained increasing popularity in various optimization tasks, since classical optimizers usually require laborious, problem-specific design and hyperparameter tuning.
ES-Based Jacobian Enables Faster Bilevel Optimization
Bilevel optimization (BO) has arisen as a powerful tool for solving many modern machine learning problems.
A Unified Off-Policy Evaluation Approach for General Value Function
We further show that unlike GTD, the learned GVFs by GenTD are guaranteed to converge to the ground truth GVFs as long as the function approximation power is sufficiently large.
Provably Faster Algorithms for Bilevel Optimization
Bilevel optimization has been widely applied in many important machine learning applications such as hyperparameter optimization and meta-learning.
Doubly Robust Off-Policy Actor-Critic: Convergence and Optimality
We also show that the overall convergence of DR-Off-PAC is doubly robust to the approximation errors that depend only on the expressive power of approximation functions.
Proximal Gradient Descent-Ascent: Variable Convergence under KŁ Geometry
By leveraging this Lyapunov function and the K{\L} geometry that parameterizes the local geometries of general nonconvex functions, we formally establish the variable convergence of proximal-GDA to a critical point $x^*$, i. e., $x_t\to x^*, y_t\to y^*(x^*)$.
Lower Bounds and Accelerated Algorithms for Bilevel Optimization
Bilevel optimization has recently attracted growing interests due to its wide applications in modern machine learning problems.
Double Q-learning: New Analysis and Sharper Finite-time Bound
Double Q-learning (Hasselt 2010) has gained significant success in practice due to its effectiveness in overcoming the overestimation issue of Q-learning.
CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guarantee
To demonstrate the theoretical performance of CRPO, we adopt natural policy gradient (NPG) for each policy update step and show that CRPO achieves an $\mathcal{O}(1/\sqrt{T})$ convergence rate to the global optimal policy in the constrained policy set and an $\mathcal{O}(1/\sqrt{T})$ error bound on constraint satisfaction.
Sample Complexity Bounds for Two Timescale Value-based Reinforcement Learning Algorithms
For linear TDC, we provide a novel non-asymptotic analysis and show that it attains an $\epsilon$-accurate solution with the optimal sample complexity of $\mathcal{O}(\epsilon^{-1}\log(1/\epsilon))$ under a constant stepsize.
Bilevel Optimization: Convergence Analysis and Enhanced Design
For the AID-based method, we orderwisely improve the previous convergence rate analysis due to a more practical parameter selection as well as a warm start strategy, and for the ITD-based method we establish the first theoretical convergence rate.
Finite-Time Analysis for Double Q-learning
Although Q-learning is one of the most successful algorithms for finding the best action-value function (and thus the optimal policy) in reinforcement learning, its implementation often suffers from large overestimation of Q-function values incurred by random sampling.
Provably Faster Algorithms for Bilevel Optimization and Applications to Meta-Learning
For the AID-based method, we orderwisely improve the previous finite-time convergence analysis due to a more practical parameter selection as well as a warm start strategy, and for the ITD-based method we establish the first theoretical convergence rate.
Enhanced First and Zeroth Order Variance Reduced Algorithms for Min-Max Optimization
Specifically, a novel variance reduction algorithm SREDA was proposed recently by (Luo et al. 2020) to solve such a problem, and was shown to achieve the optimal complexity dependence on the required accuracy level $\epsilon$.
A Primal Approach to Constrained Policy Optimization: Global Optimality and Finite-Time Analysis
To demonstrate the theoretical performance of CRPO, we adopt natural policy gradient (NPG) for each policy update step and show that CRPO achieves an $\mathcal{O}(1/\sqrt{T})$ convergence rate to the global optimal policy in the constrained policy set and an $\mathcal{O}(1/\sqrt{T})$ error bound on constraint satisfaction.
Spectral Algorithms for Community Detection in Directed Networks
Community detection in large social networks is affected by degree heterogeneity of nodes.
Momentum Q-learning with Finite-Sample Convergence Guarantee
For the infinite state-action space case, we establish the convergence guarantee for MomentumQ with linear function approximations and Markovian sampling.
Analysis of Q-learning with Adaptation and Momentum Restart for Gradient Descent
In this paper, we first characterize the convergence rate for Q-AMSGrad, which is the Q-learning algorithm with AMSGrad update (a commonly adopted alternative of Adam for theoretical analysis).
When Will Generative Adversarial Imitation Learning Algorithms Attain Global Convergence
Generative adversarial imitation learning (GAIL) is a popular inverse reinforcement learning approach for jointly optimizing policy and reward from expert trajectories.
Convergence of Meta-Learning with Task-Specific Adaptation over Partial Parameters
Although model-agnostic meta-learning (MAML) is a very successful algorithm in meta-learning practice, it can have high computational cost because it updates all model parameters over both the inner loop of task-specific adaptation and the outer-loop of meta initialization training.
Gradient Free Minimax Optimization: Variance Reduction and Faster Convergence
In this paper, we focus on such a gradient-free setting, and consider the nonconvex-strongly-concave minimax stochastic optimization problem.
Non-asymptotic Convergence Analysis of Two Time-scale (Natural) Actor-Critic Algorithms
In the first nested-loop design, actor's one update of policy is followed by an entire loop of critic's updates of the value function, and the finite-sample analysis of such AC and NAC algorithms have been recently well established.
Improving Sample Complexity Bounds for (Natural) Actor-Critic Algorithms
We show that the overall sample complexity for a mini-batch AC to attain an $\epsilon$-accurate stationary point improves the best known sample complexity of AC by an order of $\mathcal{O}(\epsilon^{-1}\log(1/\epsilon))$, and the overall sample complexity for a mini-batch NAC to attain an $\epsilon$-accurate globally optimal point improves the existing sample complexity of NAC by an order of $\mathcal{O}(\epsilon^{-1}/\log(1/\epsilon))$.
Proximal Gradient Algorithm with Momentum and Flexible Parameter Restart for Nonconvex Optimization
Our APG-restart is designed to 1) allow for adopting flexible parameter restart schemes that cover many existing ones; 2) have a global sub-linear convergence rate in nonconvex and nonsmooth optimization; and 3) have guaranteed convergence to a critical point and have various types of asymptotic convergence rates depending on the parameterization of local geometry in nonconvex and nonsmooth optimization.
Theoretical Convergence of Multi-Step Model-Agnostic Meta-Learning
As a popular meta-learning approach, the model-agnostic meta-learning (MAML) algorithm has been widely used due to its simplicity and effectiveness.
Robust Stochastic Bandit Algorithms under Probabilistic Unbounded Adversarial Attack
This paper investigates the attack model where an adversary attacks with a certain probability at each round, and its attack value can be arbitrary and unbounded if it attacks.
Non-asymptotic Convergence of Adam-type Reinforcement Learning Algorithms under Markovian Sampling
Despite the wide applications of Adam in reinforcement learning (RL), the theoretical convergence of Adam-type RL algorithms has not been established.
Reanalysis of Variance Reduced Temporal Difference Learning
Furthermore, the variance error (for both i. i. d.\ and Markovian sampling) and the bias error (for Markovian sampling) of VRTD are significantly reduced by the batch size of variance reduction in comparison to those of vanilla TD.
SpiderBoost and Momentum: Faster Variance Reduction Algorithms
SARAH and SPIDER are two recently developed stochastic variance-reduced algorithms, and SPIDER has been shown to achieve a near-optimal first-order oracle complexity in smooth nonconvex optimization.
Improved Zeroth-Order Variance Reduced Algorithms and Analysis for Nonconvex Optimization
Two types of zeroth-order stochastic algorithms have recently been designed for nonconvex optimization respectively based on the first-order techniques SVRG and SARAH/SPIDER.
History-Gradient Aided Batch Size Adaptation for Variance Reduced Algorithms
In this paper, we propose a novel scheme, which eliminates backtracking line search but still exploits the information along optimization path by adapting the batch size via history stochastic gradients.
Distributed SGD Generalizes Well Under Asynchrony
The performance of fully synchronized distributed systems has faced a bottleneck due to the big data trend, under which asynchronous distributed systems are becoming a major popularity due to their powerful scalability.
Two Time-scale Off-Policy TD Learning: Non-asymptotic Analysis over Markovian Samples
Gradient-based temporal difference (GTD) algorithms are widely used in off-policy learning scenarios.
CAN ALTQ LEARN FASTER: EXPERIMENTS AND THEORY
Differently from the popular Deep Q-Network (DQN) learning, Alternating Q-learning (AltQ) does not fully fit a target Q-function at each iteration, and is generally known to be unstable and inefficient.
Momentum Schemes with Stochastic Variance Reduction for Nonconvex Composite Optimization
In this paper, we develop novel momentum schemes with flexible coefficient settings to accelerate SPIDER for nonconvex and nonsmooth composite optimization, and show that the resulting algorithms achieve the near-optimal gradient oracle complexity for achieving a generalized first-order stationary condition.
Finite-Sample Analysis for SARSA with Linear Function Approximation
For this fitted SARSA algorithm, we also provide its finite-sample analysis.
SGD Converges to Global Minimum in Deep Learning via Star-convex Path
Stochastic gradient descent (SGD) has been found to be surprisingly effective in training a variety of deep neural networks.
MR-GAN: Manifold Regularized Generative Adversarial Networks
Despite the growing interest in generative adversarial networks (GANs), training GANs remains a challenging problem, both from a theoretical and a practical standpoint.
Minimax Estimation of Neural Net Distance
An important class of distance metrics proposed for training generative adversarial networks (GANs) is the integral probability metric (IPM), in which the neural net distance captures the practical GAN training via two neural networks.
SpiderBoost and Momentum: Faster Stochastic Variance Reduction Algorithms
SARAH and SPIDER are two recently developed stochastic variance-reduced algorithms, and SPIDER has been shown to achieve a near-optimal first-order oracle complexity in smooth nonconvex optimization.
Cubic Regularization with Momentum for Nonconvex Optimization
However, such a successful acceleration technique has not yet been proposed for second-order algorithms in nonconvex optimization. In this paper, we apply the momentum scheme to cubic regularized (CR) Newton's method and explore the potential for acceleration.
A Note on Inexact Condition for Cubic Regularized Newton's Method
This note considers the inexact cubic-regularized Newton's method (CR), which has been shown in \cite{Cartis2011a} to achieve the same order-level convergence rate to a secondary stationary point as the exact CR \citep{Nesterov2006}.
Convergence of Cubic Regularization for Nonconvex Optimization under KL Property
Cubic-regularized Newton's method (CR) is a popular algorithm that guarantees to produce a second-order stationary solution for solving nonconvex optimization problems.
K-medoids Clustering of Data Sequences with Composite Distributions
In particular, the error exponent is characterized when either the Kolmogrov-Smirnov distance or the maximum mean discrepancy are used as the distance metric.
When Will Gradient Methods Converge to Max-margin Classifier under ReLU Models?
We study the implicit bias of gradient descent methods in solving a binary classification problem over a linearly separable dataset.
Stochastic Variance-Reduced Cubic Regularization for Nonconvex Optimization
Cubic regularization (CR) is an optimization method with emerging popularity due to its capability to escape saddle points and converge to second-order stationary solutions for nonconvex optimization.
Generalization Error Bounds with Probabilistic Guarantee for SGD in Nonconvex Optimization
With strongly convex regularizers, we further establish the generalization error bounds for nonconvex loss functions under proximal SGD with high-probability guarantee, i. e., exponential concentration in probability.
Guaranteed Recovery of One-Hidden-Layer Neural Networks via Cross Entropy
We prove that with Gaussian inputs, the empirical risk based on cross entropy exhibits strong convexity and smoothness {\em uniformly} in a local neighborhood of the ground truth, as soon as the sample complexity is sufficiently large.
Critical Points of Linear Neural Networks: Analytical Forms and Landscape Properties
In this paper, we provide a necessary and sufficient characterization of the analytical forms for the critical points (as well as global minimizers) of the square loss functions for linear neural networks.
Critical Points of Neural Networks: Analytical Forms and Landscape Properties
We show that the analytical forms of the critical points characterize the values of the corresponding loss functions as well as the necessary and sufficient conditions to achieve global minimum.
Characterization of Gradient Dominance and Regularity Conditions for Neural Networks
The past decade has witnessed a successful application of deep learning to solving many challenging problems in machine learning and artificial intelligence.
Nonconvex Low-Rank Matrix Recovery with Arbitrary Outliers via Median-Truncated Gradient Descent
Recent work has demonstrated the effectiveness of gradient descent for directly recovering the factors of low-rank matrices from random linear measurements in a globally convergent manner when initialized properly.
Convergence Analysis of Proximal Gradient with Momentum for Nonconvex Optimization
Then, by exploiting the Kurdyka-{\L}ojasiewicz (\KL) property for a broad class of functions, we establish the linear and sub-linear convergence rates of the function value sequence generated by APGnc.
Reshaped Wirtinger Flow for Solving Quadratic System of Equations
In contrast to the smooth loss function used in WF, we adopt a nonsmooth but lower-order loss function, and design a gradient-like algorithm (referred to as reshaped-WF).
Reshaped Wirtinger Flow and Incremental Algorithm for Solving Quadratic System of Equations
We further develop the incremental (stochastic) reshaped Wirtinger flow (IRWF) and show that IRWF converges linearly to the true signal.
Nonparametric Detection of Geometric Structures over Networks
Sufficient conditions on minimum and maximum sizes of candidate anomalous intervals are characterized in order to guarantee the proposed test to be consistent.
Median-Truncated Nonconvex Approach for Phase Retrieval with Outliers
This paper investigates the phase retrieval problem, which aims to recover a signal from the magnitudes of its linear measurements.
Analysis of Robust PCA via Local Incoherence
We investigate the robust PCA problem of decomposing an observed matrix into the sum of a low-rank and a sparse error matrices via convex programming Principal Component Pursuit (PCP).
Nonparametric Detection of Anomalous Data Streams
samples drawn from a distribution p, whereas each anomalous sequence contains m i. i. d.
A Kernel-Based Nonparametric Test for Anomaly Detection over Line Networks
If anomalous interval does not exist, then all nodes receive samples generated by p. It is assumed that the distributions p and q are arbitrary, and are unknown.
Sharp Threshold for Multivariate Multi-Response Linear Regression via Block Regularized Lasso
The goal is to recover the support union of all regression vectors using $l_1/l_2$-regularized Lasso.
