Search Results for author:
Found 39 papers, 10 papers with code
On the Convergence of Adam under Non-uniform Smoothness: Separability from SGDM and Beyond
This paper aims to clearly distinguish between Stochastic Gradient Descent with Momentum (SGDM) and Adam in terms of their convergence rates.
Deciphering and integrating invariants for neural operator learning with various physical mechanisms
To this end, we propose Physical Invariant Attention Neural Operator (PIANO) to decipher and integrate the physical invariants (PI) for operator learning from the PDE series with various physical mechanisms.
Power-law Dynamic arising from machine learning
We study a kind of new SDE that was arisen from the research on optimization in machine learning, we call it power-law dynamic because its stationary distribution cannot have sub-Gaussian tail and obeys power-law.
O-GNN: Incorporating Ring Priors into Molecular Modeling
Despite the recent success of molecular modeling with graph neural networks (GNNs), few models explicitly take rings in compounds into consideration, consequently limiting the expressiveness of the models.
Ranked #1 on
Graph Regression
on PCQM4M-LSC
(Validation MAE metric)
NeuralStagger: Accelerating Physics-constrained Neural PDE Solver with Spatial-temporal Decomposition
Neural networks have shown great potential in accelerating the solution of partial differential equations (PDEs).
Monte Carlo Neural PDE Solver for Learning PDEs via Probabilistic Representation
To address these limitations, we propose the Monte Carlo Neural PDE Solver (MCNP Solver) for training unsupervised neural solvers via the PDEs' probabilistic representation, which regards macroscopic phenomena as ensembles of random particles.
Multilingual Speech Emotion Recognition With Multi-Gating Mechanism and Neural Architecture Search
While Speech Emotion Recognition (SER) is a common application for popular languages, it continues to be a problem for low-resourced languages, i. e., languages with no pretrained speech-to-text recognition models.
Provable Adaptivity in Adam
In particular, the existing analysis of Adam cannot clearly demonstrate the advantage of Adam over SGD.
Deep Random Vortex Method for Simulation and Inference of Navier-Stokes Equations
To this end, we propose the \emph{Deep Random Vortex Method} (DRVM), which combines the neural network with a random vortex dynamics system equivalent to the Navier-Stokes equation.
Neural Operator with Regularity Structure for Modeling Dynamics Driven by SPDEs
Stochastic partial differential equations (SPDEs) are significant tools for modeling dynamics in many areas including atmospheric sciences and physics.
Optimizing Information-theoretical Generalization Bounds via Anisotropic Noise in SGLD
We prove that with constraint to guarantee low empirical risk, the optimal noise covariance is the square root of the expected gradient covariance if both the prior and the posterior are jointly optimized.
SE(3) Equivariant Graph Neural Networks with Complete Local Frames
In this paper, we propose a framework to construct SE(3) equivariant graph neural networks that can approximate the geometric quantities efficiently.
Does Momentum Change the Implicit Regularization on Separable Data?
The momentum acceleration technique is widely adopted in many optimization algorithms.
R-Drop: Regularized Dropout for Neural Networks
Dropout is a powerful and widely used technique to regularize the training of deep neural networks.
Ranked #4 on
Machine Translation
on WMT2014 English-French
PriorGrad: Improving Conditional Denoising Diffusion Models with Data-Dependent Adaptive Prior
Denoising diffusion probabilistic models have been recently proposed to generate high-quality samples by estimating the gradient of the data density.
Incorporating NODE with Pre-trained Neural Differential Operator for Learning Dynamics
In this paper, to reduce the reliance on the numerical solver, we propose to enhance the supervised signal in the training of NODE.
Machine-Learning Non-Conservative Dynamics for New-Physics Detection
Energy conservation is a basic physics principle, the breakdown of which often implies new physics.
Optimizing Information-theoretical Generalization Bound via Anisotropic Noise of SGLD
We prove that with constraint to guarantee low empirical risk, the optimal noise covariance is the square root of the expected gradient covariance if both the prior and the posterior are jointly optimized.
UniDrop: A Simple yet Effective Technique to Improve Transformer without Extra Cost
Therefore, in this paper, we integrate different dropout techniques into the training of Transformer models.
Ranked #4 on
Machine Translation
on IWSLT2014 English-German
The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networks
Except GD, adaptive algorithms such as AdaGrad, RMSProp and Adam are popular owing to their rapid training process.
Dynamic of Stochastic Gradient Descent with State-Dependent Noise
Specifically, we show that the covariance of the noise of SGD in the local region of the local minima is a quadratic function of the state.
Interpreting Basis Path Set in Neural Networks
Based on basis path set, G-SGD algorithm significantly outperforms conventional SGD algorithm in optimizing neural networks.
Path Space for Recurrent Neural Networks with ReLU Activations
Optimization algorithms like stochastic gradient descent that optimize the neural networks in the vector space of weights, which are not positively scale-invariant.
P-BN: Towards Effective Batch Normalization in the Path Space
Hence, some new algorithms that conduct optimizations directly in the path space (the path space is proven to be PSI) were developed, such as Stochastic Gradient Descent (SGD) in the path space, and it was shown that SGD in the path space is superior to that in the weight space.
G-SGD: Optimizing ReLU Neural Networks in its Positively Scale-Invariant Space
Then, a natural question is: \emph{can we construct a new vector space that is positively scale-invariant and sufficient to represent ReLU neural networks so as to better facilitate the optimization process }?
Reinforcement Learning with Dynamic Boltzmann Softmax Updates
In this paper, we propose to update the value function with dynamic Boltzmann softmax (DBS) operator, which has good convergence property in the setting of planning and learning.
Positively Scale-Invariant Flatness of ReLU Neural Networks
That is to say, the minimum with balanced values of basis paths will more likely to be flatter and generalize better.
Expressiveness in Deep Reinforcement Learning
Based on our observations, we formally define expressiveness of the state extractor as the rank of the matrix composed by representations.
A Convergent Variant of the Boltzmann Softmax Operator in Reinforcement Learning
We then propose the dynamic Boltzmann softmax(DBS) operator to enable the convergence to the optimal value function in value iteration.
Target Transfer Q-Learning and Its Convergence Analysis
In this paper, we propose to transfer the Q-function learned in the source task to the target of the Q-learning in the new task when certain safe conditions are satisfied.
Capacity Control of ReLU Neural Networks by Basis-path Norm
Motivated by this, we propose a new norm \emph{Basis-path Norm} based on a group of linearly independent paths to measure the capacity of neural networks more accurately.
Differential Equations for Modeling Asynchronous Algorithms
Then we conduct theoretical analysis on the convergence rates of ASGD algorithm based on the continuous approximation.
$\mathcal{G}$-SGD: Optimizing ReLU Neural Networks in its Positively Scale-Invariant Space
Then, a natural question is: \emph{can we construct a new vector space that is positively scale-invariant and sufficient to represent ReLU neural networks so as to better facilitate the optimization process }?
LightGBM: A Highly Efficient Gradient Boosting Decision Tree
We prove that, since the data instances with larger gradients play a more important role in the computation of information gain, GOSS can obtain quite accurate estimation of the information gain with a much smaller data size.
Convergence Analysis of Distributed Stochastic Gradient Descent with Shuffling
First, we give a mathematical formulation for the practical data processing procedure in distributed machine learning, which we call data partition with global/local shuffling.
A Communication-Efficient Parallel Algorithm for Decision Tree
After partitioning the training data onto a number of (e. g., $M$) machines, this algorithm performs both local voting and global voting in each iteration.
Asynchronous Stochastic Proximal Optimization Algorithms with Variance Reduction
The results verified our theoretical findings and demonstrated the practical efficiency of the asynchronous stochastic proximal algorithms with variance reduction.
Generalization Error Bounds for Optimization Algorithms via Stability
Many machine learning tasks can be formulated as Regularized Empirical Risk Minimization (R-ERM), and solved by optimization algorithms such as gradient descent (GD), stochastic gradient descent (SGD), and stochastic variance reduction (SVRG).
Asynchronous Stochastic Gradient Descent with Delay Compensation
We propose a novel technology to compensate this delay, so as to make the optimization behavior of ASGD closer to that of sequential SGD.
