Search Results for author:
Found 51 papers, 11 papers with code
Shuffling Momentum Gradient Algorithm for Convex Optimization
The Stochastic Gradient Descent method (SGD) and its stochastic variants have become methods of choice for solving finite-sum optimization problems arising from machine learning and data science thanks to their ability to handle large-scale applications and big datasets.
On Partial Optimal Transport: Revising the Infeasibility of Sinkhorn and Efficient Gradient Methods
This paper studies the Partial Optimal Transport (POT) problem between two unbalanced measures with at most $n$ supports and its applications in various AI tasks such as color transfer or domain adaptation.
One step closer to unbiased aleatoric uncertainty estimation
Neural networks are powerful tools in various applications, and quantifying their uncertainty is crucial for reliable decision-making.
A Supervised Contrastive Learning Pretrain-Finetune Approach for Time Series
Foundation models have recently gained attention within the field of machine learning thanks to its efficiency in broad data processing.
Correlated Attention in Transformers for Multivariate Time Series
Multivariate time series (MTS) analysis prevails in real-world applications such as finance, climate science and healthcare.
Promoting Robustness of Randomized Smoothing: Two Cost-Effective Approaches
The second approach introduces a post-processing method EsbRS which greatly improves the robustness certificate based on building model ensembles.
Batch Clipping and Adaptive Layerwise Clipping for Differential Private Stochastic Gradient Descent
In this paper, we propose {\em a new ALC and provide rigorous DP proofs for both BC and ALC}.
Learning Robust and Consistent Time Series Representations: A Dilated Inception-Based Approach
Representation learning for time series has been an important research area for decades.
An End-to-End Time Series Model for Simultaneous Imputation and Forecast
Time series forecasting using historical data has been an interesting and challenging topic, especially when the data is corrupted by missing values.
Label-Free Concept Bottleneck Models
Motivated by these challenges, we propose Label-free CBM which is a novel framework to transform any neural network into an interpretable CBM without labeled concept data, while retaining a high accuracy.
ConCerNet: A Contrastive Learning Based Framework for Automated Conservation Law Discovery and Trustworthy Dynamical System Prediction
Deep neural networks (DNN) have shown great capacity of modeling a dynamical system; nevertheless, they usually do not obey physics constraints such as conservation laws.
Generalizing DP-SGD with Shuffling and Batch Clipping
Classical differential private DP-SGD implements individual clipping with random subsampling, which forces a mini-batch SGD approach.
Finding Optimal Policy for Queueing Models: New Parameterization
In this work, we investigate the optimization aspects of the queueing model as a RL environment and provide insight to learn the optimal policy efficiently.
On Unbalanced Optimal Transport: Gradient Methods, Sparsity and Approximation Error
In this paper, we propose a novel algorithm based on Gradient Extrapolation Method (GEM-UOT) to find an $\varepsilon$-approximate solution to the UOT problem in $O\big( \kappa \log\big(\frac{\tau n}{\varepsilon}\big) \big)$ iterations with $\widetilde{O}(n^2)$ per-iteration cost, where $\kappa$ is the condition number depending on only the two input measures.
Finite-Sum Optimization: A New Perspective for Convergence to a Global Solution
How and under what assumptions is guaranteed convergence to a \textit{global} minimum possible?
Attacking c-MARL More Effectively: A Data Driven Approach
In recent years, a proliferation of methods were developed for cooperative multi-agent reinforcement learning (c-MARL).
Nesterov Accelerated Shuffling Gradient Method for Convex Optimization
This rate is better than that of any other shuffling gradient methods in convex regime.
Interpretable Clustering via Multi-Polytope Machines
To solve our formulation we propose a two phase approach where we first initialize clusters and polytopes using alternating minimization, and then use coordinate descent to boost clustering performance.
On the Equivalence between Neural Network and Support Vector Machine
Therefore, in this work, we propose to establish the equivalence between NN and SVM, and specifically, the infinitely wide NN trained by soft margin loss and the standard soft margin SVM with NTK trained by subgradient descent.
Ensembling Graph Predictions for AMR Parsing
In many machine learning tasks, models are trained to predict structure data such as graphs.
Ranked #2 on
AMR Parsing
on LDC2020T02
(using extra training data)
Tactics on Refining Decision Boundary for Improving Certification-based Robust Training
In verification-based robust training, existing methods utilize relaxation based methods to bound the worst case performance of neural networks given certain perturbation.
New Perspective on the Global Convergence of Finite-Sum Optimization
How and under what assumptions is guaranteed convergence to a \textit{global} minimum possible?
Evaluating Robustness of Cooperative MARL
In recent years, a proliferation of methods were developed for multi-agent reinforcement learning (MARL).
FedDR -- Randomized Douglas-Rachford Splitting Algorithms for Nonconvex Federated Composite Optimization
These new algorithms can handle statistical and system heterogeneity, which are the two main challenges in federated learning, while achieving the best known communication complexity.
Proactive DP: A Multple Target Optimization Framework for DP-SGD
Generally, DP-SGD is $(\epsilon\leq 1/2,\delta=1/N)$-DP if $\sigma=\sqrt{2(\epsilon +\ln(1/\delta))/\epsilon}$ with $T$ at least $\approx 2k^2/\epsilon$ and $(2/e)^2k^2-1/2\geq \ln(N)$, where $T$ is the total number of rounds, and $K=kN$ is the total number of gradient computations where $k$ measures $K$ in number of epochs of size $N$ of the local data set.
Buffer Zone based Defense against Adversarial Examples in Image Classification
Based on our study of these defenses, we develop three contributions.
SMG: A Shuffling Gradient-Based Method with Momentum
When the shuffling strategy is fixed, we develop another new algorithm that is similar to existing momentum methods, and prove the same convergence rates for this algorithm under the $L$-smoothness and bounded gradient assumptions.
A Scalable MIP-based Method for Learning Optimal Multivariate Decision Trees
Several recent publications report advances in training optimal decision trees (ODT) using mixed-integer programs (MIP), due to algorithmic advances in integer programming and a growing interest in addressing the inherent suboptimality of heuristic approaches such as CART.
Hogwild! over Distributed Local Data Sets with Linearly Increasing Mini-Batch Sizes
We consider big data analysis where training data is distributed among local data sets in a heterogeneous way -- and we wish to move SGD computations to local compute nodes where local data resides.
An Optimal Hybrid Variance-Reduced Algorithm for Stochastic Composite Nonconvex Optimization
In this note we propose a new variant of the hybrid variance-reduced proximal gradient method in [7] to solve a common stochastic composite nonconvex optimization problem under standard assumptions.
Asynchronous Federated Learning with Reduced Number of Rounds and with Differential Privacy from Less Aggregated Gaussian Noise
The feasibility of federated learning is highly constrained by the server-clients infrastructure in terms of network communication.
Hybrid Variance-Reduced SGD Algorithms For Nonconvex-Concave Minimax Problems
This problem class has several computational challenges due to its nonsmoothness, nonconvexity, nonlinearity, and non-separability of the objective functions.
Finite-Time Analysis of Stochastic Gradient Descent under Markov Randomness
Motivated by broad applications in reinforcement learning and machine learning, this paper considers the popular stochastic gradient descent (SGD) when the gradients of the underlying objective function are sampled from Markov processes.
A Hybrid Stochastic Policy Gradient Algorithm for Reinforcement Learning
We propose a novel hybrid stochastic policy gradient estimator by combining an unbiased policy gradient estimator, the REINFORCE estimator, with another biased one, an adapted SARAH estimator for policy optimization.
A Unified Convergence Analysis for Shuffling-Type Gradient Methods
We also study uniformly randomized shuffling variants with different learning rates and model assumptions.
Stochastic Gauss-Newton Algorithms for Nonconvex Compositional Optimization
In the expectation case, we establish $\mathcal{O}(\varepsilon^{-2})$ iteration-complexity to achieve a stationary point in expectation and estimate the total number of stochastic oracle calls for both function value and its Jacobian, where $\varepsilon$ is a desired accuracy.
BUZz: BUffer Zones for defending adversarial examples in image classification
We argue that our defense based on buffer zones offers significant improvements over state-of-the-art defenses.
A Hybrid Stochastic Optimization Framework for Stochastic Composite Nonconvex Optimization
We introduce a new approach to develop stochastic optimization algorithms for a class of stochastic composite and possibly nonconvex optimization problems.
Hybrid Stochastic Gradient Descent Algorithms for Stochastic Nonconvex Optimization
We introduce a hybrid stochastic estimator to design stochastic gradient algorithms for solving stochastic optimization problems.
ProxSARAH: An Efficient Algorithmic Framework for Stochastic Composite Nonconvex Optimization
We also specify the algorithm to the non-composite case that covers existing state-of-the-arts in terms of complexity bounds.
Finite-Sum Smooth Optimization with SARAH
The total complexity (measured as the total number of gradient computations) of a stochastic first-order optimization algorithm that finds a first-order stationary point of a finite-sum smooth nonconvex objective function $F(w)=\frac{1}{n} \sum_{i=1}^n f_i(w)$ has been proven to be at least $\Omega(\sqrt{n}/\epsilon)$ for $n \leq \mathcal{O}(\epsilon^{-2})$ where $\epsilon$ denotes the attained accuracy $\mathbb{E}[ \|\nabla F(\tilde{w})\|^2] \leq \epsilon$ for the outputted approximation $\tilde{w}$ (Fang et al., 2018).
DTN: A Learning Rate Scheme with Convergence Rate of $\mathcal{O}(1/t)$ for SGD
This paper has some inconsistent results, i. e., we made some failed claims because we did some mistakes for using the test criterion for a series.
PROVEN: Certifying Robustness of Neural Networks with a Probabilistic Approach
With deep neural networks providing state-of-the-art machine learning models for numerous machine learning tasks, quantifying the robustness of these models has become an important area of research.
Inexact SARAH Algorithm for Stochastic Optimization
We develop and analyze a variant of the SARAH algorithm, which does not require computation of the exact gradient.
New Convergence Aspects of Stochastic Gradient Algorithms
We show the convergence of SGD for strongly convex objective function without using bounded gradient assumption when $\{\eta_t\}$ is a diminishing sequence and $\sum_{t=0}^\infty \eta_t \rightarrow \infty$.
Tight Dimension Independent Lower Bound on the Expected Convergence Rate for Diminishing Step Sizes in SGD
We study the convergence of Stochastic Gradient Descent (SGD) for strongly convex objective functions.
Characterization of Convex Objective Functions and Optimal Expected Convergence Rates for SGD
We study Stochastic Gradient Descent (SGD) with diminishing step sizes for convex objective functions.
SGD and Hogwild! Convergence Without the Bounded Gradients Assumption
In (Bottou et al., 2016), a new analysis of convergence of SGD is performed under the assumption that stochastic gradients are bounded with respect to the true gradient norm.
When Does Stochastic Gradient Algorithm Work Well?
In this paper, we consider a general stochastic optimization problem which is often at the core of supervised learning, such as deep learning and linear classification.
Stochastic Recursive Gradient Algorithm for Nonconvex Optimization
In this paper, we study and analyze the mini-batch version of StochAstic Recursive grAdient algoritHm (SARAH), a method employing the stochastic recursive gradient, for solving empirical loss minimization for the case of nonconvex losses.
SARAH: A Novel Method for Machine Learning Problems Using Stochastic Recursive Gradient
In this paper, we propose a StochAstic Recursive grAdient algoritHm (SARAH), as well as its practical variant SARAH+, as a novel approach to the finite-sum minimization problems.
