Search Results for author:
Found 73 papers, 18 papers with code
Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling
We conduct a systematic study of the approximation properties of Transformer for sequence modeling with long, sparse and complicated memory.
Anchor function: a type of benchmark functions for studying language models
However, language model research faces significant challenges, especially for academic research groups with constrained resources.
Machine-Learned Invertible Coarse Graining for Multiscale Molecular Modeling
Multiscale molecular modeling is widely applied in scientific research of molecular properties over large time and length scales.
MAC: A unified framework boosting low resource automatic speech recognition
Furthermore, in the ASR task, MAC beats wav2vec2 (with fine-tuning) on common voice datasets of Cantonese and gets really competitive results on common voice datasets of Taiwanese and Japanese.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
A multi-scale sampling method for accurate and robust deep neural network to predict combustion chemical kinetics
The current work aims to understand two basic questions regarding the deep neural network (DNN) method: what data the DNN needs and how general the DNN method can be.
A deep learning-based model reduction (DeePMR) method for simplifying chemical kinetics
The mechanism reduction is modeled as an optimization problem on Boolean space, where a Boolean vector, each entry corresponding to a species, represents a reduced mechanism.
DeePN$^2$: A deep learning-based non-Newtonian hydrodynamic model
A long standing problem in the modeling of non-Newtonian hydrodynamics of polymeric flows is the availability of reliable and interpretable hydrodynamic models that faithfully encode the underlying micro-scale polymer dynamics.
DeepHAM: A Global Solution Method for Heterogeneous Agent Models with Aggregate Shocks
An efficient, reliable, and interpretable global solution method, the Deep learning-based algorithm for Heterogeneous Agent Models (DeepHAM), is proposed for solving high dimensional heterogeneous agent models with aggregate shocks.
Generalization Error of GAN from the Discriminator's Perspective
The generative adversarial network (GAN) is a well-known model for learning high-dimensional distributions, but the mechanism for its generalization ability is not understood.
MOD-Net: A Machine Learning Approach via Model-Operator-Data Network for Solving PDEs
In this paper, we propose a a machine learning approach via model-operator-data network (MOD-Net) for solving PDEs.
An $L^2$ Analysis of Reinforcement Learning in High Dimensions with Kernel and Neural Network Approximation
Reinforcement learning (RL) algorithms based on high-dimensional function approximation have achieved tremendous empirical success in large-scale problems with an enormous number of states.
The Phase Diagram of a Deep Potential Water Model
Using the Deep Potential methodology, we construct a model that reproduces accurately the potential energy surface of the SCAN approximation of density functional theory for water, from low temperature and pressure to about 2400 K and 50 GPa, excluding the vapor stability region.
Chemical Physics
On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers
A recent numerical study observed that neural network classifiers enjoy a large degree of symmetry in the penultimate layer.
Some observations on high-dimensional partial differential equations with Barron data
We use explicit representation formulas to show that solutions to certain partial differential equations lie in Barron spaces or multilayer spaces if the PDE data lie in such function spaces.
Generalization and Memorization: The Bias Potential Model
Models for learning probability distributions such as generative models and density estimators behave quite differently from models for learning functions.
A deep learning-based ODE solver for chemical kinetics
Besides, the ignition delay time differences are within 1%.
Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learning
The result shows that (1) the escaping time of both SGD and ADAM~depends on the Radon measure of the basin positively and the heaviness of gradient noise negatively; (2) for the same basin, SGD enjoys smaller escaping time than ADAM, mainly because (a) the geometry adaptation in ADAM~via adaptively scaling each gradient coordinate well diminishes the anisotropic structure in gradient noise and results in larger Radon measure of a basin; (b) the exponential gradient average in ADAM~smooths its gradient and leads to lighter gradient noise tails than SGD.
The Knowledge Graph for Macroeconomic Analysis with Alternative Big Data
The current knowledge system of macroeconomics is built on interactions among a small number of variables, since traditional macroeconomic models can mostly handle a handful of inputs.
Interpretable Neural Networks for Panel Data Analysis in Economics
In this paper, we propose a class of interpretable neural network models that can achieve both high prediction accuracy and interpretability.
A priori estimates for classification problems using neural networks
We consider binary and multi-class classification problems using hypothesis classes of neural networks.
Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don't
The purpose of this article is to review the achievements made in the last few years towards the understanding of the reasons behind the success and subtleties of neural network-based machine learning.
On the Curse of Memory in Recurrent Neural Networks: Approximation and Optimization Analysis
We study the approximation properties and optimization dynamics of recurrent neural networks (RNNs) when applied to learn input-output relationships in temporal data.
A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithms
The dynamic behavior of RMSprop and Adam algorithms is studied through a combination of careful numerical experiments and theoretical explanations.
OnsagerNet: Learning Stable and Interpretable Dynamics using a Generalized Onsager Principle
We further apply this method to study Rayleigh-Benard convection and learn Lorenz-like low dimensional autonomous reduced order models that capture both qualitative and quantitative properties of the underlying dynamics.
The Slow Deterioration of the Generalization Error of the Random Feature Model
The random feature model exhibits a kind of resonance behavior when the number of parameters is close to the training sample size.
On the Banach spaces associated with multi-layer ReLU networks: Function representation, approximation theory and gradient descent dynamics
The key to this work is a new way of representing functions in some form of expectations, motivated by multi-layer neural networks.
Coarse-grained spectral projection (CGSP): a deep learning-assisted approach to quantum unitary dynamics
We propose the coarse-grained spectral projection method (CGSP), a deep learning-assisted approach for tackling quantum unitary dynamic problems with an emphasis on quench dynamics.
The Quenching-Activation Behavior of the Gradient Descent Dynamics for Two-layer Neural Network Models
A numerical and phenomenological study of the gradient descent (GD) algorithm for training two-layer neural network models is carried out for different parameter regimes when the target function can be accurately approximated by a relatively small number of neurons.
Representation formulas and pointwise properties for Barron functions
We study the natural function space for infinitely wide two-layer neural networks with ReLU activation (Barron space) and establish different representation formulae.
Deep Potential generation scheme and simulation protocol for the Li10GeP2S12-type superionic conductors
It has been a challenge to accurately simulate Li-ion diffusion processes in battery materials at room temperature using {\it ab initio} molecular dynamics (AIMD) due to its high computational cost.
Computational Physics Materials Science Chemical Physics
Integrating Machine Learning with Physics-Based Modeling
Machine learning is poised as a very powerful tool that can drastically improve our ability to carry out scientific research.
Kolmogorov Width Decay and Poor Approximators in Machine Learning: Shallow Neural Networks, Random Feature Models and Neural Tangent Kernels
We establish a scale separation of Kolmogorov width type between subspaces of a given Banach space under the condition that a sequence of linear maps converges much faster on one of the subspaces.
Can Shallow Neural Networks Beat the Curse of Dimensionality? A mean field training perspective
Thus gradient descent training for fitting reasonably smooth, but truly high-dimensional data may be subject to the curse of dimensionality.
Pushing the limit of molecular dynamics with ab initio accuracy to 100 million atoms with machine learning
For 35 years, {\it ab initio} molecular dynamics (AIMD) has been the method of choice for modeling complex atomistic phenomena from first principles.
Computational Physics
Machine learning based non-Newtonian fluid model with molecular fidelity
We introduce a machine-learning-based framework for constructing continuum non-Newtonian fluid dynamics model directly from a micro-scale description.
Machine Learning from a Continuous Viewpoint
We demonstrate that conventional machine learning models and algorithms, such as the random feature model, the two-layer neural network model and the residual neural network model, can all be recovered (in a scaled form) as particular discretizations of different continuous formulations.
The Generalization Error of the Minimum-norm Solutions for Over-parameterized Neural Networks
We study the generalization properties of minimum-norm solutions for three over-parametrized machine learning models including the random feature model, the two-layer neural network model and the residual network model.
DP-GEN: A concurrent learning platform for the generation of reliable deep learning based potential energy models
Materials 3, 023804] and is capable of generating uniformly accurate deep learning based PES models in a way that minimizes human intervention and the computational cost for data generation and model training.
Computational Physics
A mathematical model for universal semantics
We characterize the meaning of words with language-independent numerical fingerprints, through a mathematical analysis of recurring patterns in texts.
Deep neural network for Wannier function centers
We introduce a deep neural network (DNN) model that assigns the position of the centers of the electronic charge in each atomic configuration on a molecular dynamics trajectory.
Computational Physics Materials Science Chemical Physics
The Barron Space and the Flow-induced Function Spaces for Neural Network Models
We define the Barron space and show that it is the right space for two-layer neural network models in the sense that optimal direct and inverse approximation theorems hold for functions in the Barron space.
Monge-Amp\`ere Flow for Generative Modeling
We present a deep generative model, named Monge-Amp\`ere flow, which builds on continuous-time gradient flow arising from the Monge-Amp\`ere equation in optimal transport theory.
A Priori Estimates of the Generalization Error for Two-layer Neural Networks
These new estimates are a priori in nature in the sense that the bounds depend only on some norms of the underlying functions to be fitted, not the parameters in the model.
Analysis of the Gradient Descent Algorithm for a Deep Neural Network Model with Skip-connections
In addition, it is also shown that the GD path is uniformly close to the functions given by the related random feature model.
A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics
In the over-parametrized regime, it is shown that gradient descent dynamics can achieve zero training loss exponentially fast regardless of the quality of the labels.
A Priori Estimates of the Population Risk for Residual Networks
An important part of the regularized model is the usage of a new path norm, called the weighted path norm, as the regularization term.
How SGD Selects the Global Minima in Over-parameterized Learning: A Dynamical Stability Perspective
The question of which global minima are accessible by a stochastic gradient decent (SGD) algorithm with specific learning rate and batch size is studied from the perspective of dynamical stability.
Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundations
We develop the mathematical foundations of the stochastic modified equations (SME) framework for analyzing the dynamics of stochastic gradient algorithms, where the latter is approximated by a class of stochastic differential equations with small noise parameters.
Active Learning of Uniformly Accurate Inter-atomic Potentials for Materials Simulation
An active learning procedure called Deep Potential Generator (DP-GEN) is proposed for the construction of accurate and transferable machine learning-based models of the potential energy surface (PES) for the molecular modeling of materials.
A Priori Estimates of the Population Risk for Two-layer Neural Networks
New estimates for the population risk are established for two-layer neural networks.
Monge-Ampère Flow for Generative Modeling
We present a deep generative model, named Monge-Amp\`ere flow, which builds on continuous-time gradient flow arising from the Monge-Amp\`ere equation in optimal transport theory.
Model Reduction with Memory and the Machine Learning of Dynamical Systems
The well-known Mori-Zwanzig theory tells us that model reduction leads to memory effect.
Solving Many-Electron Schrödinger Equation Using Deep Neural Networks
We introduce a new family of trial wave-functions based on deep neural networks to solve the many-electron Schr\"odinger equation.
Computational Physics Chemical Physics
A Mean-Field Optimal Control Formulation of Deep Learning
This paper introduces the mathematical formulation of the population risk minimization problem in deep learning as a mean-field optimal control problem.
Exponential Convergence of the Deep Neural Network Approximation for Analytic Functions
We prove that for analytic functions in low dimension, the convergence rate of the deep neural network approximation is exponential.
End-to-end Symmetry Preserving Inter-atomic Potential Energy Model for Finite and Extended Systems
Machine learning models are changing the paradigm of molecular modeling, which is a fundamental tool for material science, chemistry, and computational biology.
Computational Physics Materials Science Chemical Physics
Understanding and Enhancing the Transferability of Adversarial Examples
State-of-the-art deep neural networks are known to be vulnerable to adversarial examples, formed by applying small but malicious perturbations to the original inputs.
Enhancing the Transferability of Adversarial Examples with Noise Reduced Gradient
Deep neural networks provide state-of-the-art performance for many applications of interest.
DeePMD-kit: A deep learning package for many-body potential energy representation and molecular dynamics
Here we describe DeePMD-kit, a package written in Python/C++ that has been designed to minimize the effort required to build deep learning based representation of potential energy and force field and to perform molecular dynamics.
Reinforced dynamics for enhanced sampling in large atomic and molecular systems
Like metadynamics, it allows for an efficient exploration of the configuration space by adding an adaptively computed biasing potential to the original dynamics.
Maximum Principle Based Algorithms for Deep Learning
The continuous dynamical system approach to deep learning is explored in order to devise alternative frameworks for training algorithms.
The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problems
We propose a deep learning based method, the Deep Ritz Method, for numerically solving variational problems, particularly the ones that arise from partial differential equations.
Machine learning approximation algorithms for high-dimensional fully nonlinear partial differential equations and second-order backward stochastic differential equations
The PDEs in such applications are high-dimensional as the dimension corresponds to the number of financial assets in a portfolio.
Deep Potential Molecular Dynamics: a scalable model with the accuracy of quantum mechanics
We introduce a scheme for molecular simulations, the Deep Potential Molecular Dynamics (DeePMD) method, based on a many-body potential and interatomic forces generated by a carefully crafted deep neural network trained with ab initio data.
Solving high-dimensional partial differential equations using deep learning
Developing algorithms for solving high-dimensional partial differential equations (PDEs) has been an exceedingly difficult task for a long time, due to the notoriously difficult problem known as the "curse of dimensionality".
Deep Potential: a general representation of a many-body potential energy surface
When tested on a wide variety of examples, Deep Potential is able to reproduce the original model, whether empirical or quantum mechanics based, within chemical accuracy.
Computational Physics
Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes
It is widely observed that deep learning models with learned parameters generalize well, even with much more model parameters than the number of training samples.
Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations
We propose a new algorithm for solving parabolic partial differential equations (PDEs) and backward stochastic differential equations (BSDEs) in high dimension, by making an analogy between the BSDE and reinforcement learning with the gradient of the solution playing the role of the policy function, and the loss function given by the error between the prescribed terminal condition and the solution of the BSDE.
Deep Learning Approximation for Stochastic Control Problems
Many real world stochastic control problems suffer from the "curse of dimensionality".
Stochastic modified equations and adaptive stochastic gradient algorithms
We develop the method of stochastic modified equations (SME), in which stochastic gradient algorithms are approximated in the weak sense by continuous-time stochastic differential equations.
Convolutional neural networks with low-rank regularization
Recently, tensor decompositions have been used for speeding up CNNs.
Functional Frank-Wolfe Boosting for General Loss Functions
Yet, as the number of base hypotheses becomes larger, boosting can lead to a deterioration of test performance.
Multiscale Adaptive Representation of Signals: I. The Basic Framework
The new framework, called AdaFrame, improves over dictionary learning-based techniques in terms of computational efficiency at inference time.
