Search Results for author:
Found 49 papers, 10 papers with code
On the Generalization Ability of Unsupervised Pretraining
Recent advances in unsupervised learning have shown that unsupervised pre-training, followed by fine-tuning, can improve model generalization.
On the Generalization Capability of Temporal Graph Learning Algorithms: Theoretical Insights and a Simpler Method
Temporal Graph Learning (TGL) has become a prevalent technique across diverse real-world applications, especially in domains where data can be represented as a graph and evolves over time.
Stochastic Quantum Sampling for Non-Logconcave Distributions and Estimating Partition Functions
We also incorporate a stochastic gradient oracle that implements the quantum walk operators inexactly by only using mini-batch gradients.
Understanding Deep Gradient Leakage via Inversion Influence Functions
Deep Gradient Leakage (DGL) is a highly effective attack that recovers private training images from gradient vectors.
On the Hardness of Robustness Transfer: A Perspective from Rademacher Complexity over Symmetric Difference Hypothesis Space
Recent studies demonstrated that the adversarially robust learning under $\ell_\infty$ attack is harder to generalize to different domains than standard domain adaptation.
Do We Really Need Complicated Model Architectures For Temporal Networks?
Recurrent neural network (RNN) and self-attention mechanism (SAM) are the de facto methods to extract spatial-temporal information for temporal graph learning.
Efficiently Forgetting What You Have Learned in Graph Representation Learning via Projection
As privacy protection receives much attention, unlearning the effect of a specific node from a pre-trained graph learning model has become equally important.
Tight Analysis of Extra-gradient and Optimistic Gradient Methods For Nonconvex Minimax Problems
Despite the established convergence theory of Optimistic Gradient Descent Ascent (OGDA) and Extragradient (EG) methods for the convex-concave minimax problems, little is known about the theoretical guarantees of these methods in nonconvex settings.
Learning Distributionally Robust Models at Scale via Composite Optimization
To train machine learning models that are robust to distribution shifts in the data, distributionally robust optimization (DRO) has been proven very effective.
DyFormer: A Scalable Dynamic Graph Transformer with Provable Benefits on Generalization Ability
To achieve efficient and scalable training, we propose temporal-union graph structure and its associated subgraph-based node sampling strategy.
Learn Locally, Correct Globally: A Distributed Algorithm for Training Graph Neural Networks
To solve the performance degradation, we propose to apply $\text{{Global Server Corrections}}$ on the server to refine the locally learned models.
On Provable Benefits of Depth in Training Graph Convolutional Networks
Graph Convolutional Networks (GCNs) are known to suffer from performance degradation as the number of layers increases, which is usually attributed to over-smoothing.
Meta-learning with an Adaptive Task Scheduler
In ATS, for the first time, we design a neural scheduler to decide which meta-training tasks to use next by predicting the probability being sampled for each candidate task, and train the scheduler to optimize the generalization capacity of the meta-model to unseen tasks.
Local SGD Optimizes Overparameterized Neural Networks in Polynomial Time
This work is the first to show the convergence of Local SGD on non-smooth functions, and will shed lights on the optimization theory of federated training of deep neural networks.
Pareto Efficient Fairness in Supervised Learning: From Extraction to Tracing
The proposed PEF notion is definition-agnostic, meaning that any well-defined notion of fairness can be reduced to the PEF notion.
On the Importance of Sampling in Training GCNs: Tighter Analysis and Variance Reduction
In this paper, we describe and analyze a general doubly variance reduction schema that can accelerate any sampling method under the memory budget.
Distributionally Robust Federated Averaging
To compensate for this, we propose a Distributionally Robust Federated Averaging (DRFA) algorithm that employs a novel snapshotting scheme to approximate the accumulation of history gradients of the mixing parameter.
Local Stochastic Gradient Descent Ascent: Convergence Analysis and Communication Efficiency
Local SGD is a promising approach to overcome the communication overhead in distributed learning by reducing the synchronization frequency among worker nodes.
Communication-efficient k-Means for Edge-based Machine Learning
We consider the problem of computing the k-means centers for a large high-dimensional dataset in the context of edge-based machine learning, where data sources offload machine learning computation to nearby edge servers.
On the Importance of Sampling in Training GCNs: Convergence Analysis and Variance Reduction
In this paper, we describe and analyze a general \textbf{\textit{doubly variance reduction}} schema that can accelerate any sampling method under the memory budget.
GCN meets GPU: Decoupling “When to Sample” from “How to Sample”
Sampling-based methods promise scalability improvements when paired with stochastic gradient descent in training Graph Convolutional Networks (GCNs).
Online Structured Meta-learning
When a new task is encountered, it constructs a meta-knowledge pathway by either utilizing the most relevant knowledge blocks or exploring new blocks.
Targeted Data-driven Regularization for Out-of-Distribution Generalization
The proposed framework, named targeted data-driven regularization (TDR), is model- and dataset-agnostic and employs a target dataset that resembles the desired nature of test data in order to guide the learning process in a coupled manner.
Federated Learning with Compression: Unified Analysis and Sharp Guarantees
In federated learning, communication cost is often a critical bottleneck to scale up distributed optimization algorithms to collaboratively learn a model from millions of devices with potentially unreliable or limited communication and heterogeneous data distributions.
Minimal Variance Sampling with Provable Guarantees for Fast Training of Graph Neural Networks
In this paper, we theoretically analyze the variance of sampling methods and show that, due to the composite structure of empirical risk, the variance of any sampling method can be decomposed into \textit{embedding approximation variance} in the forward stage and \textit{stochastic gradient variance} in the backward stage that necessities mitigating both types of variance to obtain faster convergence rate.
Adaptive Personalized Federated Learning
Investigation of the degree of personalization in federated learning algorithms has shown that only maximizing the performance of the global model will confine the capacity of the local models to personalize.
Efficient Fair Principal Component Analysis
It has been shown that dimension reduction methods such as PCA may be inherently prone to unfairness and treat data from different sensitive groups such as race, color, sex, etc., unfairly.
On the Convergence of Local Descent Methods in Federated Learning
To bridge this gap, we demonstrate that by properly analyzing the effect of unbiased gradients and sampling schema in federated setting, under mild assumptions, the implicit variance reduction feature of local distributed methods generalize to heterogeneous data shards and exhibits the best known convergence rates of homogeneous setting both in general nonconvex and under {\pl}~ condition (generalization of strong-convexity).
Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization
Specifically, we show that for loss functions that satisfy the Polyak-{\L}ojasiewicz condition, $O((pT)^{1/3})$ rounds of communication suffice to achieve a linear speed up, that is, an error of $O(1/pT)$, where $T$ is the total number of model updates at each worker.
Trading Redundancy for Communication: Speeding up Distributed SGD for Non-convex Optimization
Communication overhead is one of the key challenges that hinder the scalability of distributed optimization algorithms to train large neural networks.
Learning Feature Nonlinearities with Non-Convex Regularized Binned Regression
Evaluations on synthetic and real datasets demonstrate that algorithm is competitive with current state-of-the-art and accurately learns feature nonlinearities.
Sketching Meets Random Projection in the Dual: A Provable Recovery Algorithm for Big and High-dimensional Data
Sketching techniques have become popular for scaling up machine learning algorithms by reducing the sample size or dimensionality of massive data sets, while still maintaining the statistical power of big data.
Smooth and Strong: MAP Inference with Linear Convergence
Maximum a-posteriori (MAP) inference is an important task for many applications.
Train and Test Tightness of LP Relaxations in Structured Prediction
Structured prediction is used in areas such as computer vision and natural language processing to predict structured outputs such as segmentations or parse trees.
Matrix Factorization with Explicit Trust and Distrust Relationships
With the advent of online social networks, recommender systems have became crucial for the success of many online applications/services due to their significance role in tailoring these applications to user-specific needs or preferences.
Exploiting Smoothness in Statistical Learning, Sequential Prediction, and Stochastic Optimization
In the last several years, the intimate connection between convex optimization and learning problems, in both statistical and sequential frameworks, has shifted the focus of algorithmic machine learning to examine this interplay.
Binary Excess Risk for Smooth Convex Surrogates
In statistical learning theory, convex surrogates of the 0-1 loss are highly preferred because of the computational and theoretical virtues that convexity brings in.
Excess Risk Bounds for Exponentially Concave Losses
The overarching goal of this paper is to derive excess risk bounds for learning from exp-concave loss functions in passive and sequential learning settings.
Mixed Optimization for Smooth Functions
It is well known that the optimal convergence rate for stochastic optimization of smooth functions is $[O(1/\sqrt{T})]$, which is same as stochastic optimization of Lipschitz continuous convex functions.
Stochastic Convex Optimization with Multiple Objectives
It leverages on the theory of Lagrangian method in constrained optimization and attains the optimal convergence rate of $[O(1/ \sqrt{T})]$ in high probability for general Lipschitz continuous objectives.
Linear Convergence with Condition Number Independent Access of Full Gradients
For smooth and strongly convex optimization, the optimal iteration complexity of the gradient-based algorithm is $O(\sqrt{\kappa}\log 1/\epsilon)$, where $\kappa$ is the conditional number.
Beating the Minimax Rate of Active Learning with Prior Knowledge
Under the assumption that the norm of the optimal classifier that minimizes the convex risk is available, our analysis shows that the introduction of the convex surrogate loss yields an exponential reduction in the label complexity even when the parameter $\kappa$ of the Tsybakov noise is larger than $1$.
MixedGrad: An O(1/T) Convergence Rate Algorithm for Stochastic Smooth Optimization
It is well known that the optimal convergence rate for stochastic optimization of smooth functions is $O(1/\sqrt{T})$, which is same as stochastic optimization of Lipschitz continuous convex functions.
Passive Learning with Target Risk
In this paper we consider learning in passive setting but with a slight modification.
Stochastic Gradient Descent with Only One Projection
Although many variants of stochastic gradient descent have been proposed for large-scale convex optimization, most of them require projecting the solution at {\it each} iteration to ensure that the obtained solution stays within the feasible domain.
Nyström Method vs Random Fourier Features: A Theoretical and Empirical Comparison
Both random Fourier features and the Nyström method have been successfully applied to efficient kernel learning.
Online Stochastic Optimization with Multiple Objectives
We first propose a projection based algorithm which attains an $O(T^{-1/3})$ convergence rate.
Recovering the Optimal Solution by Dual Random Projection
Random projection has been widely used in data classification.
An Efficient Primal-Dual Prox Method for Non-Smooth Optimization
We study the non-smooth optimization problems in machine learning, where both the loss function and the regularizer are non-smooth functions.
